Skip to content

Logo for BC Open Textbooks

Toggle Menu

Search in book:

Search in book …

SEARCH

CONTENTS

RESEARCH METHODS IN PSYCHOLOGY

Chapter 5: Psychological Measurement

Reliability and Validity of Measurement

Learning Objectives

Define reliability, including the different types and how they are assessed.

Define validity, including the different types and how they are assessed.

Describe the kinds of evidence that would be relevant to assessing the reliability and validity of a
particular measure.

Again, measurement involves assigning scores to individuals so that they represent some characteristic
of the individuals. But how do researchers know that the scores actually represent the characteristic,
especially when it is a construct like intelligence, self-esteem, depression, or working memory capacity?
The answer is that they conduct research using the measure to confirm that the scores make sense
based on their understanding of the construct being measured. This is an extremely important point.
Psychologists do not simply assume that their measures work. Instead, they collect data to demonstrate
that they work. If their research does not demonstrate that a measure works, they stop using it.

As an informal example, imagine that you have been dieting for a month. Your clothes seem to be fitting
more loosely, and several friends have asked if you have lost weight. If at this point your bathroom scale
indicated that you had lost 10 pounds, this would make sense and you would continue to use the scale.
But if it indicated that you had gained 10 pounds, you would rightly conclude that it was broken and
either fix it or get rid of it. In evaluating a measurement method, psychologists consider two general
dimensions: reliability and validity.

Reliability

Reliability refers to the consistency of a measure. Psychologists consider three types of consistency: over
time (test-retest reliability), across items (internal consistency), and across different researchers (inter-
rater reliability).

Test-Retest Reliability

When researchers measure a construct that they assume to be consistent across time, then the scores
they obtain should also be consistent across time. Test-retest reliability is the extent to which this is
actually the case. For example, intelligence is generally thought to be consistent across time. A person
who is highly intelligent today will be highly intelligent next week. This means that any good measure of
intelligence should produce roughly the same scores for this individual next week as it does today.
Clearly, a measure that produces highly inconsistent scores over time cannot be a very good measure of
a construct that is supposed to be consistent.

Assessing test-retest reliability requires using the measure on a group of people at one time, using it
again on the same group of people at a later time, and then looking at test-retest correlation between
the two sets of scores. This is typically done by graphing the data in a scatterplot and computing
Pearson’s r. Figure 5.2 shows the correlation between two sets of scores of several university students
on the Rosenberg Self-Esteem Scale, administered two times, a week apart. Pearson’s r for these data is
+.95. In general, a test-retest correlation of +.80 or greater is considered to indicate good reliability.

Score at time 1 is on the x-axis and score at time 2 is on the y-axis, showing fairly consistent scores

Figure 5.2 Test-Retest Correlation Between Two Sets of Scores of Several College Students on the
Rosenberg Self-Esteem Scale, Given Two Times a Week Apart

Again, high test-retest correlations make sense when the construct being measured is assumed to be
consistent over time, which is the case for intelligence, self-esteem, and the Big Five personality
dimensions. But other constructs are not assumed to be stable over time. The very nature of mood, for
example, is that it changes. So a measure of mood that produced a low test-retest correlation over a
period of a month would not be a cause for concern.
Internal Consistency

A second kind of reliability is internal consistency, which is the consistency of people’s responses across
the items on a multiple-item measure. In general, all the items on such measures are supposed to reflect
the same underlying construct, so people’s scores on those items should be correlated with each other.
On the Rosenberg Self-Esteem Scale, people who agree that they are a person of worth should tend to
agree that that they have a number of good qualities. If people’s responses to the different items are
not correlated with each other, then it would no longer make sense to claim that they are all measuring
the same underlying construct. This is as true for behavioural and physiological measures as for self-
report measures. For example, people might make a series of bets in a simulated game of roulette as a
measure of their level of risk seeking. This measure would be internally consistent to the extent that
individual participants’ bets were consistently high or low across trials.

Like test-retest reliability, internal consistency can only be assessed by collecting and analyzing data.
One approach is to look at a split-half correlation. This involves splitting the items into two sets, such as
the first and second halves of the items or the even- and odd-numbered items. Then a score is
computed for each set of items, and the relationship between the two sets of scores is examined. For
example, Figure 5.3 shows the split-half correlation between several university students’ scores on the
even-numbered items and their scores on the odd-numbered items of the Rosenberg Self-Esteem Scale.
Pearson’s r for these data is +.88. A split-half correlation of +.80 or greater is generally considered good
internal consistency.

Score on even-numbered items is on the x-axis and score on odd-numbered items is on the y-axis,
showing fairly consistent scores

Figure 5.3 Split-Half Correlation Between Several College Students’ Scores on the Even-Numbered Items
and Their Scores on the Odd-Numbered Items of the Rosenberg Self-Esteem Scale

Perhaps the most common measure of internal consistency used by researchers in psychology is a
statistic called Cronbach’s α (the Greek letter alpha). Conceptually, α is the mean of all possible split-half
correlations for a set of items. For example, there are 252 ways to split a set of 10 items into two sets of
five. Cronbach’s α would be the mean of the 252 split-half correlations. Note that this is not how α is
actually computed, but it is a correct way of interpreting the meaning of this statistic. Again, a value of
+.80 or greater is generally taken to indicate good internal consistency.

Interrater Reliability

Many behavioural measures involve significant judgment on the part of an observer or a rater. Inter-
rater reliability is the extent to which different observers are consistent in their judgments. For example,
if you were interested in measuring university students’ social skills, you could make video recordings of
them as they interacted with another student whom they are meeting for the first time. Then you could
have two or more observers watch the videos and rate each student’s level of social skills. To the extent
that each participant does in fact have some level of social skills that can be detected by an attentive
observer, different observers’ ratings should be highly correlated with each other. Inter-rater reliability
would also have been measured in Bandura’s Bobo doll study. In this case, the observers’ ratings of how
many acts of aggression a particular child committed while playing with the Bobo doll should have been
highly positively correlated. Interrater reliability is often assessed using Cronbach’s α when the
judgments are quantitative or an analogous statistic called Cohen’s κ (the Greek letter kappa) when they
are categorical.

Validity

Validity is the extent to which the scores from a measure represent the variable they are intended to.
But how do researchers make this judgment? We have already considered one factor that they take into
account—reliability. When a measure has good test-retest reliability and internal consistency,
researchers should be more confident that the scores represent what they are supposed to. There has
to be more to it, however, because a measure can be extremely reliable but have no validity
whatsoever. As an absurd example, imagine someone who believes that people’s index finger length
reflects their self-esteem and therefore tries to measure self-esteem by holding a ruler up to people’s
index fingers. Although this measure would have extremely good test-retest reliability, it would have
absolutely no validity. The fact that one person’s index finger is a centimetre longer than another’s
would indicate nothing about which one had higher self-esteem.

Discussions of validity usually divide it into several distinct “types.” But a good way to interpret these
types is that they are other kinds of evidence—in addition to reliability—that should be taken into
account when judging the validity of a measure. Here we consider three basic kinds: face validity,
content validity, and criterion validity.

Face Validity

Face validity is the extent to which a measurement method appears “on its face” to measure the
construct of interest. Most people would expect a self-esteem questionnaire to include items about
whether they see themselves as a person of worth and whether they think they have good qualities. So
a questionnaire that included these kinds of items would have good face validity. The finger-length
method of measuring self-esteem, on the other hand, seems to have nothing to do with self-esteem and
therefore has poor face validity. Although face validity can be assessed quantitatively—for example, by
having a large sample of people rate a measure in terms of whether it appears to measure what it is
intended to—it is usually assessed informally.

Face validity is at best a very weak kind of evidence that a measurement method is measuring what it is
supposed to. One reason is that it is based on people’s intuitions about human behaviour, which are
frequently wrong. It is also the case that many established measures in psychology work quite well
despite lacking face validity. The Minnesota Multiphasic Personality Inventory-2 (MMPI-2) measures
many personality characteristics and disorders by having people decide whether each of over 567
different statements applies to them—where many of the statements do not have any obvious
relationship to the construct that they measure. For example, the items “I enjoy detective or mystery
stories” and “The sight of blood doesn’t frighten me or make me sick” both measure the suppression of
aggression. In this case, it is not the participants’ literal answers to these questions that are of interest,
but rather whether the pattern of the participants’ responses to a series of questions matches those of
individuals who tend to suppress their aggression.

Content Validity

Content validity is the extent to which a measure “covers” the construct of interest. For example, if a
researcher conceptually defines test anxiety as involving both sympathetic nervous system activation
(leading to nervous feelings) and negative thoughts, then his measure of test anxiety should include
items about both nervous feelings and negative thoughts. Or consider that attitudes are usually defined
as involving thoughts, feelings, and actions toward something. By this conceptual definition, a person
has a positive attitude toward exercise to the extent that he or she thinks positive thoughts about
exercising, feels good about exercising, and actually exercises. So to have good content validity, a
measure of people’s attitudes toward exercise would have to reflect all three of these aspects. Like face
validity, content validity is not usually assessed quantitatively. Instead, it is assessed by carefully
checking the measurement method against the conceptual definition of the construct.

Criterion Validity

Criterion validity is the extent to which people’s scores on a measure are correlated with other variables
(known as criteria) that one would expect them to be correlated with. For example, people’s scores on a
new measure of test anxiety should be negatively correlated with their performance on an important
school exam. If it were found that people’s scores were in fact negatively correlated with their exam
performance, then this would be a piece of evidence that these scores really represent people’s test
anxiety. But if it were found that people scored equally well on the exam regardless of their test anxiety
scores, then this would cast doubt on the validity of the measure.
A criterion can be any variable that one has reason to think should be correlated with the construct
being measured, and there will usually be many of them. For example, one would expect test anxiety
scores to be negatively correlated with exam performance and course grades and positively correlated
with general anxiety and with blood pressure during an exam. Or imagine that a researcher develops a
new measure of physical risk taking. People’s scores on this measure should be correlated with their
participation in “extreme” activities such as snowboarding and rock climbing, the number of speeding
tickets they have received, and even the number of broken bones they have had over the years. When
the criterion is measured at the same time as the construct, criterion validity is referred to as concurrent
validity; however, when the criterion is measured at some point in the future (after the construct has
been measured), it is referred to as predictive validity (because scores on the measure have “predicted”
a future outcome).

Criteria can also include other measures of the same construct. For example, one would expect new
measures of test anxiety or physical risk taking to be positively correlated with existing measures of the
same constructs. This is known as convergent validity.

Assessing convergent validity requires collecting data using the measure. Researchers John Cacioppo
and Richard Petty did this when they created their self-report Need for Cognition Scale to measure how
much people value and engage in thinking (Cacioppo & Petty, 1982)[1]. In a series of studies, they
showed that people’s scores were positively correlated with their scores on a standardized academic
achievement test, and that their scores were negatively correlated with their scores on a measure of
dogmatism (which represents a tendency toward obedience). In the years since it was created, the Need
for Cognition Scale has been used in literally hundreds of studies and has been shown to be correlated
with a wide variety of other variables, including the effectiveness of an advertisement, interest in
politics, and juror decisions (Petty, Briñol, Loersch, & McCaslin, 2009)[2].

Discriminant Validity

Discriminant validity, on the other hand, is the extent to which scores on a measure are not correlated
with measures of variables that are conceptually distinct. For example, self-esteem is a general attitude
toward the self that is fairly stable over time. It is not the same as mood, which is how good or bad one
happens to be feeling right now. So people’s scores on a new measure of self-esteem should not be very
highly correlated with their moods. If the new measure of self-esteem were highly correlated with a
measure of mood, it could be argued that the new measure is not really measuring self-esteem; it is
measuring mood instead.
When they created the Need for Cognition Scale, Cacioppo and Petty also provided evidence of
discriminant validity by showing that people’s scores were not correlated with certain other variables.
For example, they found only a weak correlation between people’s need for cognition and a measure of
their cognitive style—the extent to which they tend to think analytically by breaking ideas into smaller
parts or holistically in terms of “the big picture.” They also found no correlation between people’s need
for cognition and measures of their test anxiety and their tendency to respond in socially desirable ways.
All these low correlations provide evidence that the measure is reflecting a conceptually distinct
construct.

Key Takeaways

Psychological researchers do not simply assume that their measures work. Instead, they conduct
research to show that they work. If they cannot show that they work, they stop using them.

There are two distinct criteria by which researchers evaluate their measures: reliability and validity.
Reliability is consistency across time (test-retest reliability), across items (internal consistency), and
across researchers (interrater reliability). Validity is the extent to which the scores actually represent the
variable they are intended to.

Validity is a judgment based on various types of evidence. The relevant evidence includes the measure’s
reliability, whether it covers the construct of interest, and whether the scores it produces are correlated
with other variables they are expected to be correlated with and not correlated with variables that are
conceptually distinct.

The reliability and validity of a measure is not established by any single study but by the pattern of
results across multiple studies. The assessment of reliability and validity is an ongoing process.

Exercises

Practice: Ask several friends to complete the Rosenberg Self-Esteem Scale. Then assess its internal
consistency by making a scatterplot to show the split-half correlation (even- vs. odd-numbered items).
Compute Pearson’s r too if you know how.

Discussion: Think back to the last college exam you took and think of the exam as a psychological
measure. What construct do you think it was intended to measure? Comment on its face and content
validity. What data could you collect to assess its reliability and criterion validity?

Cacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal of Personality and Social
Psychology, 42, 116–131. ↵
Petty, R. E, Briñol, P., Loersch, C., & McCaslin, M. J. (2009). The need for cognition. In M. R. Leary & R. H.
Hoyle (Eds.), Handbook of individual differences in social behaviour (pp. 318–329). New York, NY:
Guilford Press. ↵

Previous: Understanding Psychological Measurement

Next: Practical Strategies for Psychological Measurement

BACK TO TOP

LICENSE

Icon for the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

Research Methods in Psychology by Paul C. Price, Rajiv Jhangiani, & I-Chant A. Chiang is licensed under a
Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where
otherwise noted.

SHARE THIS BOOK

Powered by Pressbooks

Guides and Tutorials

YouTube Pressbooks on YouTube

Pressbooks on Twitter

Reliability and validity

The reliability of an assessment tool is the extent to which it measures learning consistently.

The validity of an assessment tool is the extent by which it measures what it was designed to measure.

Reliability
The reliability of an assessment tool is the extent to which it consistently and accurately measures
learning.

When the results of an assessment are reliable, we can be confident that repeated or equivalent
assessments will provide consistent results. This puts us in a better position to make generalised
statements about a student’s level of achievement, which is especially important when we are using the
results of an assessment to make decisions about teaching and learning, or when we are reporting back
to students and their parents or caregivers. No results, however, can be completely reliable. There is
always some random variation that may affect the assessment, so educators should always be prepared
to question results.

Factors which can affect reliability:

The length of the assessment – a longer assessment generally produces more reliable results.

The suitability of the questions or tasks for the students being assessed.

The phrasing and terminology of the questions.

The consistency in test administration – for example, the length of time given for the assessment,
instructions given to students before the test.

The design of the marking schedule and moderation of marking procedures.

The readiness of students for the assessment – for example, a hot afternoon or straight after physical
activity might not be the best time for students to be assessed.

How to be sure that a formal assessment tool is reliable

Check in the user manual for evidence of the reliability coefficient. These are measured between zero
and 1. A coefficient of 0.9 or more indicates a high degree of reliability.

Assessment tool manuals contain comprehensive administration guidelines. It is essential to read the
manual thoroughly before conducting the assessment.

Validity
Educational assessment should always have a clear purpose. Nothing will be gained from assessment
unless the assessment has some validity for the purpose. For that reason, validity is the most important
single attribute of a good test.

The validity of an assessment tool is the extent to which it measures what it was designed to measure,
without contamination from other characteristics. For example, a test of reading comprehension should
not require mathematical ability.

There are several different types of validity:

Face validity: do the assessment items appear to be appropriate?

Content validity: does the assessment content cover what you want to assess?

Criterion-related validity: how well does the test measure what you want it to?

Construct validity: are you measuring what you think you're measuring?

It is fairly obvious that a valid assessment should have a good coverage of the criteria (concepts, skills
and knowledge) relevant to the purpose of the examination. The important notion here is the purpose.
For example:

The PROBE test is a form of reading running record which measures reading behaviours and includes
some comprehension questions. It allows teachers to see the reading strategies that students are using,
and potential problems with decoding. The test would not, however, provide in-depth information
about a student’s comprehension strategies across a range of texts.

STAR (Supplementary Test of Achievement in Reading) is not designed as a comprehensive test of

reading ability. It focuses on assessing students’ vocabulary understanding, basic sentence
comprehension and paragraph comprehension. It is most appropriately used for students who don’t
score well on more general testing (such as PAT or e-asTTle) as it provides a more fine grained analysis
of basic comprehension strategies.

There is an important relationship between reliability and validity. An assessment that has very low
reliability will also have low validity; clearly a measurement with very poor accuracy or consistency is
unlikely to be fit for its purpose. But, by the same token, the things required to achieve a very high
degree of reliability can impact negatively on validity. For example, consistency in assessment conditions
leads to greater reliability because it reduces 'noise' (variability) in the results. On the other hand, one of
the things that can improve validity is flexibility in assessment tasks and conditions. Such flexibility
allows assessment to be set appropriate to the learning context and to be made relevant to particular
groups of students. Insisting on highly consistent assessment conditions to attain high reliability will
result in little flexibility, and might therefore limit validity.

The Overall Teacher Judgment balances these notions with a balance between the reliability of a formal
assessment tool, and the flexibility to use other evidence to make a judgment.

The Graide Network

SEPTEMBER 10, 2018

Importance of Validity and Reliability in Classroom Assessments

DATA IN SCHOOLS

DOWNLOAD PDF VERSION

Validity and Reliability in Classroom Assessments

Pop Quiz:

One of the following tests is reliable but not valid and the other is valid but not reliable. Can you figure
out which is which?

You want to measure student intelligence so you ask students to do as many push-ups as they can every
day for a week.

You want to measure students’ perception of their teacher using a survey but the teacher hands out the
evaluations right after she reprimands her class, which she doesn’t normally do.

Continue reading to find out the answer--and why it matters so much.

Validity and Reliability in Education

Schools all over the country are beginning to develop a culture of data, which is the integration of data
into the day-to-day operations of a school in order to achieve classroom, school, and district-wide goals.
One of the biggest difficulties that comes with this integration is determining what data will provide an
accurate reflection of those goals.

Such considerations are particularly important when the goals of the school aren’t put into terms that
lend themselves to cut and dry analysis; school goals often describe the improvement of abstract
concepts like “school climate.”

Schools interested in establishing a culture of data are advised to come up with a plan before going off
to collect it. They need to first determine what their ultimate goal is and what achievement of that goal
looks like. An understanding of the definition of success allows the school to ask focused questions to
help measure that success, which may be answered with the data.

For example, if a school is interested in increasing literacy, one focused question might ask: which
groups of students are consistently scoring lower on standardized English tests? If a school is interested
in promoting a strong climate of inclusiveness, a focused question may be: do teachers treat different
types of students unequally?

These focused questions are analogous to research questions asked in academic fields such as
psychology, economics, and, unsurprisingly, education. However, the question itself does not always
indicate which instrument (e.g. a standardized test, student survey, etc.) is optimal.

If the wrong instrument is used, the results can quickly become meaningless or uninterpretable, thereby
rendering them inadequate in determining a school’s standing in or progress toward their goals.

Reliability and Validity

Differences Between Validity and Reliability

When creating a question to quantify a goal, or when deciding on a data instrument to secure the
results to that question, two concepts are universally agreed upon by researchers to be of pique
importance.
These two concepts are called validity and reliability, and they refer to the quality and accuracy of data
instruments.

WHAT IS VALIDITY?

The validity of an instrument is the idea that the instrument measures what it intends to measure.

Validity pertains to the connection between the purpose of the research and which data the researcher
chooses to quantify that purpose.

For example, imagine a researcher who decides to measure the intelligence of a sample of students.
Some measures, like physical strength, possess no natural connection to intelligence. Thus, a test of
physical strength, like how many push-ups a student could do, would be an invalid test of intelligence.

What is Validity

WHAT IS RELIABILITY?

Reliability, on the other hand, is not at all concerned with intent, instead asking whether the test used to
collect data produces accurate results. In this context, accuracy is defined by consistency (whether the
results could be replicated).

The property of ignorance of intent allows an instrument to be simultaneously reliable and invalid.

Returning to the example above, if we measure the number of pushups the same students can do every
day for a week (which, it should be noted, is not long enough to significantly increase strength) and each
person does approximately the same amount of pushups on each day, the test is reliable. But, clearly,
the reliability of these results still does not render the number of pushups per student a valid measure
of intelligence.

Because reliability does not concern the actual relevance of the data in answering a focused question,
validity will generally take precedence over reliability. Moreover, schools will often assess two levels of
validity:
the validity of the research question itself in quantifying the larger, generally more abstract goal

the validity of the instrument chosen to answer the research question

See the diagram below as an example:

Difference Between Reliability and Validity

Although reliability may not take center stage, both properties are important when trying to achieve any
goal with the help of data. So how can schools implement them? In research, reliability and validity are
often computed with statistical programs. However, even for school leaders who may not have the
resources to perform proper statistical analysis, an understanding of these concepts will still allow for
intuitive examination of how their data instruments hold up, thus affording them the opportunity to
formulate better assessments to achieve educational goals. So, let’s dive a little deeper.

A Deeper Look at Validity

The most basic definition of validity is that an instrument is valid if it measures what it intends to
measure. It’s easier to understand this definition through looking at examples of invalidity. Colin Foster,
an expert in mathematics education at the University of Nottingham, gives the example of a reading test
meant to measure literacy that is given in a very small font size. A highly literate student with bad
eyesight may fail the test because they can’t physically read the passages supplied. Thus, such a test
would not be a valid measure of literacy (though it may be a valid measure of eyesight). Such an
example highlights the fact that validity is wholly dependent on the purpose behind a test. More
generally, in a study plagued by weak validity, “it would be possible for someone to fail the test situation
rather than the intended test subject.” Validity can be divided into several different categories, some of
which relate very closely to one another. We will discuss a few of the most relevant categories in the
following paragraphs.

Types of Validity

Types of Validity

WHAT IS CONSTRUCT VALIDITY?

Construct validity refers to the general idea that the realization of a theory should be aligned with the
theory itself. If this sounds like the broader definition of validity, it’s because construct validity is viewed
by researchers as “a unifying concept of validity” that encompasses other forms, as opposed to a
completely separate type.

It is not always cited in the literature, but, as Drew Westen and Robert Rosenthal write in “Quantifying
Construct Validity: Two Simple Measures,” construct validity “is at the heart of any study in which
researchers use a measure as an index of a variable that is itself not directly observable.”

The ability to apply concrete measures to abstract concepts is obviously important to researchers who
are trying to measure concepts like intelligence or kindness. However, it also applies to schools, whose
goals and objectives (and therefore what they intend to measure) are often described using broad terms
like “effective leadership” or “challenging instruction.”

Construct validity ensures the interpretability of results, thereby paving the way for effective and
efficient data-based decision making by school leaders.

What is Construct Validity

WHAT IS CRITERION VALIDITY?

Criterion validity refers to the correlation between a test and a criterion that is already accepted as a
valid measure of the goal or question. If a test is highly correlated with another valid criterion, it is more
likely that the test is also valid.

Criterion validity tends to be measured through statistical computations of correlation coefficients,

although it’s possible that existing research has already determined the validity of a particular test that
schools want to collect data on.

WHAT IS CONTENT VALIDITY?

Content validity refers to the actual content within a test. A test that is valid in content should
adequately examine all aspects that define the objective.
Content validity is not a statistical measurement, but rather a qualitative one. For example, a
standardized assessment in 9th-grade biology is content-valid if it covers all topics taught in a standard
9th-grade biology course.

Warren Schillingburg, an education specialist and associate superintendent, advises that determination
of content-validity “should include several teachers (and content experts when possible) in evaluating
how well the test represents the content taught.”

While this advice is certainly helpful for academic tests, content validity is of particular importance when
the goal is more abstract, as the components of that goal are more subjective.

School inclusiveness, for example, may not only be defined by the equality of treatment across student
groups, but by other factors, such as equal opportunities to participate in extracurricular activities.

Despite its complexity, the qualitative nature of content validity makes it a particularly accessible
measure for all school leaders to take into consideration when creating data instruments.
Types of Validity

A CASE STUDY ON VALIDITY

To understand the different types of validity and how they interact, consider the example of Baltimore
Public Schools trying to measure school climate.

School climate is a broad term, and its intangible nature can make it difficult to determine the validity of
tests that attempt to quantify it. Baltimore Public Schools found research from The National Center for
School Climate (NCSC) which set out five criterion that contribute to the overall health of a school’s
climate. These criteria are safety, teaching and learning, interpersonal relationships, environment, and
leadership, which the paper also defines on a practical level.

Because the NCSC’s criterion were generally accepted as valid measures of school climate, Baltimore
City Schools sought to find tools that “are aligned with the domains and indicators proposed by the
National School Climate Center.” This is essentially asking whether the tools Baltimore City Schools used
were criterion-valid measures of school climate.

Baltimore City Schools introduced four data instruments, predominantly surveys, to find valid measures
of school climate based on these criterion. They found that “each source addresses different school
climate domains with varying emphasis,” implying that the usage of one tool may not yield content-valid
results, but that the usage of all four “can be construed as complementary parts of the same larger
picture.” Thus, sometimes validity can be achieved by using multiple tools from multiple viewpoints.

Reliability in Education

A Deeper Look at Reliability

TYPES OF RELIABILITY

The reliability of an assessment refers to the consistency of results. The most basic interpretation
generally references something called test-retest reliability, which is characterized by the replicability of
results. That is to say, if a group of students takes a test twice, both the results for individual students,
as well as the relationship among students’ results, should be similar across tests.
However, there are two other types of reliability: alternate-form and internal consistency. Alternate
form is a measurement of how test scores compare across two similar assessments given in a short time
frame. Alternate form similarly refers to the consistency of both individual scores and positional
relationships. Internal consistency is analogous to content validity and is defined as a measure of how
the actual content of an assessment works together to evaluate understanding of a concept.

LIMITATIONS OF RELIABILITY

The three types of reliability work together to produce, according to Schillingburg, “confidence… that
the test score earned is a good representation of a child’s actual knowledge of the content.” Reliability is
important in the design of assessments because no assessment is truly perfect. A test produces an
estimate of a student’s “true” score, or the score the student would receive if given a perfect test;
however, due to imperfect design, tests can rarely, if ever, wholly capture that score. Thus, tests should
aim to be reliable, or to get as close to that true score as possible.

Imperfect testing is not the only issue with reliability. Reliability is sensitive to the stability of extraneous
influences, such as a student’s mood. Extraneous influences could be particularly dangerous in the
collection of perceptions data, or data that measures students, teachers, and other members of the
community’s perception of the school, which is often used in measurements of school culture and
climate.

Uncontrollable changes in external factors could influence how a respondent perceives their
environment, making an otherwise reliable instrument seem unreliable. For example, if a student or
class is reprimanded the day that they are given a survey to evaluate their teacher, the evaluation of the
teacher may be uncharacteristically negative. The same survey given a few days later may not yield the
same results. However, most extraneous influences relevant to students tend to occur on an individual
level, and therefore are not a major concern in the reliability of data for larger samples.

How to Improve Reliability

HOW TO IMPROVE RELIABILITY

On the other hand, extraneous influences relevant to other agents in the classroom could affect the
scores of an entire class.
If the grader of an assessment is sensitive to external factors, their given grades may reflect this
sensitivity, therefore making the results unreliable. Assessments that go beyond cut-and-dry responses
engender a responsibility for the grader to review the consistency of their results.

Some of this variability can be resolved through the use of clear and specific rubrics for grading an
assessment. Rubrics limit the ability of any grader to apply normative criteria to their grading, thereby
controlling for the influence of grader biases. However, rubrics, like tests, are imperfect tools and care
must be taken to ensure reliable results.

How does one ensure reliability? Measuring the reliability of assessments is often done with statistical
computations.

The three measurements of reliability discussed above all have associated coefficients that standard
statistical packages will calculate. However, schools that don’t have access to such tools shouldn’t simply
throw caution to the wind and abandon these concepts when thinking about data.

Schillingburg advises that at the classroom level, educators can maintain reliability by:

Creating clear instructions for each assignment

Writing questions that capture the material taught

Seeking feedback regarding the clarity and thoroughness of the assessment from students and
colleagues.

With such care, the average test given in a classroom will be reliable. Moreover, if any errors in
reliability arise, Schillingburg assures that class-level decisions made based on unreliable data are
generally reversible, e.g. assessments found to be unreliable may be rewritten based on feedback
provided.
However, reliability, or the lack thereof, can create problems for larger-scale projects, as the results of
these assessments generally form the basis for decisions that could be costly for a school or district to
either implement or reverse.

Reliability in the Classroom

Conclusion

Validity and reliability are meaningful measurements that should be taken into account when
attempting to evaluate the status of or progress toward any objective a district, school, or classroom
has.

If precise statistical measurements of these properties are not able to be made, educators should
attempt to evaluate the validity and reliability of data through intuition, previous research, and
collaboration as much as possible.

An understanding of validity and reliability allows educators to make decisions that improve the lives of
their students both academically and socially, as these concepts teach educators how to quantify the
abstract goals their school or district has set.

To learn more about how The Graide Network can help your school meet its goals, check out our
information page here.

Download PDF Version

BACK TO TOP

CELEBRATING A MILESTONE

IN THE NEWS

EXPLORE SAMPLE REPORTS

Copyright © 2020 The Graide Network | The Chicago Literacy Alliance at 641 W Lake St, Chicago, IL,
60661 | Privacy Policy & Terms of Use
AP® and Advanced Placement® are trademarks registered by the College Board, which is not affiliated
with, and does not endorse, this website.

Icons made by Freepik from www.flaticon.com