Session 4

Specific Objectives:

Learning Content:
 Objectives
1. Understand what is meant by reliability
2. Be able to evaluate the reliability of a
measure
3. Understand what is meant by validity
4. Be able to evaluate the validity of a
measure
 What is Reliability?
• Reliability is:
othe consistency of your
measurement instrument
othe degree to which an instrument
measures the same way each time
it is used under the same
condition with the same subjects
 What in the world is a
measurement instrument?
• Any tool that you use to measure with…
• What “instrument” might you use to measure
the following items?
a scale
How heavy the apples are

a measuring cup
How much orange juice there is

a yardstick
How tall the wall is
 More on Instruments
What would the following instruments measure?

The size of
someone’s foot

The perfect amount

Eyesight of spaghetti!
 Wechsler Scale- an instrument that
measures the Intelligence Quotient (I.Q.)
 More on Instruments
Often there are many different instruments that could be
used to measure the same thing.
Take distance for example…

A car odometer Or walk it with a

Measure it out on a map
You could use a pedometer
measuring tape

You can count paces Use a distance wheel

 Factors in choosing an instrument
• Ease of use – do you know how to use it?
A high-tech pedometer cannot be used by everyone.
• Access – do you have access to the instrument?
Where do you get a distance wheel from? What if you don’t have a car
– and therefore no odometer?
• Appropriateness – is it the best tool for what you are
measuring?
You wouldn’t use a ruler to measure the distance from here to Manila;
or you wouldn’t use a car odometer to measure the length of a room.
• Accuracy – how precise do your measurements have
to be?
Counting paces is easy and cheap but it is only an estimate.
• Cost – how expensive is it to use the instrument or
interpret the results?
You can also use a laser but that can be really expensive.
 Back to Reliability…
• Reliability is:
othe consistency of your
measurement instrument
othe degree to which an instrument
measures the same way each time
it is used under the same
condition with the same subjects
 Reliability
• Imagine that you are using a ruler
to measure a book What do you think would happen if you
waited 10 minutes and measured the
book again, how long would it be then?
…Probably still 7 inches

What if you spun the ruler around!

And shook it up really good?!
And it measures about 7 inches across
Now what would it say?
…Probably still 7 inches
 Reliability
• Your ruler…
o was consistent
o measured the same way each time it was used
under the same condition with the same
object

The book did not change and therefore the ruler

reported back the same measurement

Your ruler is RELIABLE

•
Reliability
Every time the water begins to boil
the kettle whistles
• Every time it is 6:30 in the morning,
the alarm clock goes on

• Every time the nurse All of these tools

uses the little plastic of measurement
cup, it has 2 are reliable given
tablespoons of the same
medicine temperature, time
and volume, they
measure the same
consistently.
 •
Reliability
Reliability alone does not mean that you have a good
instrument however.
• Imagine the following reliable instrument:
One day an elephant
got on Margie’s scale
Every morning Margie and it still read 165
gets on the scale and 0 lbs.!
165
every morning it reads
165 lbs. How is this scale
It seems pretty reliable reliable?
since Margie hasn’t It is reliable with
gained or lost any Margie, but it fails
weight. to measure a
different subject
 Reliability
• What about this reliable instrument…
This clock reads 6:15
If nothing changes – if time stands
still, will the clock still say the same
thing?
YES! It’s very reliable. You always
know exactly what it is going to say.
The problem is, even if time moves
the clock will not move…but it is
still reliable.
This brings us to
Validity
 What is Validity?
• Validity asks
oif an instrument measures what it
is supposed to
ohow “true” or accurate the
measurement is
Reliable but not Valid
165
These instruments are very
RELIABLE
They both report consistently – too
consistently
But, neither measures what it is
supposed to:
• The scale is not really measuring
weight
• The clock is not measuring time
They are NOT VALID
 Reliable but not Valid
• Remember our reliable ruler?

Can it measure how how full the

loud the radio is? glass is?
The ruler may be reliable (and
perhaps even valid) but not in all
how smart the
situations! It is only valid for
girl is?
measuring length.
 Putting Reliability and Validity
Together
• Every instrument can be evaluated on two
dimensions:
o Reliability
• How consistent it is given the same
conditions
o Validity
• If it measures what it is supposed to
and how accurate it is
 Putting Reliability and Validity
Together
• Imagine that we have 3 fish tank
thermometers, a blue one, a red
one, and a green one.

•The blue one always reads the same temperature no

matter how hot or cold the water is.
•The red one shows a different temperature every
time even if I just measured it 5 seconds earlier.
•The green one seems to read accurately, warm when
the water is warm and cold when the water is cool.
 Complete the chart below
Is it measuring Is it reliable?
Is it consistent? what it is Is it valid?
supposed to?
Blue (always
reads the same Reliable but
temperature no Yes No Not valid
matter what)
Red (different
temperature every
time even if Not reliable
No No
nothing has Not valid
changed)
Green (warm Yes, the
when the water is thermometer only
warm and cold changes if the Reliable and
Yes
when the water is temperature Valid
cool) changes
 What can be said of the reliability and
validity of the following?
• A spelling test with the following item: 2 + 5 = ____
o Probably reliable, if you get it wrong once you will probably get it
wrong again (assuming no new learning) – same with getting it right.
o Lacks validity, this is more appropriate for a math test, not a spelling
test
• An elastic ruler (every time you use it is stretches to a different length)
o Lacks reliability
o You can’t have validity without reliability
• A thermometer used to measure volume
o Probably reliable
o Lacks validity for this task
• A scale that reads 40 pounds at baseline
o Reliable, will consistently be 40 lbs. off
o Not valid
 Consider a Maze Task to
measure fine motor skills
Is it reliable? – if you do the maze
once, do you think you need the
same amount of time to finish it a
second time?
You’ll probably be faster the
second time just from practicing –
even though your motor skills
didn’t change any.
Is it valid? We’ll say it lacks reliability.
No, a measure must be reliable in order to be valid.
It is probably a more valid measure of memory.
©
Consider the World Geography quiz below:
1. The nickname for the state of New Hampshire is: Is it reliable? – if you don’t
a. Vacationland
b. The Pine Tree State know the answers, are you
c. The Granite State going to suddenly know them
2. The most populated U.S. city is: tomorrow or next week?
a. New York Probably not.
b. Los Angeles
c. Chicago We’ll say it’s reliable
3. The "Coyote" State is: Is it valid?
a. New Mexico
b. Nebraska This doesn’t look like a measure of
c. South Dakota World Geography, it focuses on
4. The largest state is: America. It doesn’t even seem like
a. Texas American geography – maybe
b. Alaska
c. Hawaii American Geographical Trivia.
5. The highest mountain is the U.S. is: It’s not a valid measure of World
a. Mt. McKinley
b. Mt. Shasta Geography.
c. Mt. Logan
 Thinking More about
Validity
Validity is whether or not the instrument
measures what it is designed to measure.

Below are three constructs that you we use

to evaluate the validity of a measure:

• Face Validity
• Predictive Validity
• Concurrent Validity
 Three Validities
• Face Validity -- Do the questions look like they
measure what they are supposed to?

• Predictive Validity -- Do you imagine that this

measure would predict something that it
logically should?

• Concurrent Validity – Do you think this

measure correlates strongly with something that
it logically should?
 Face Validity
Do the questions look like they measure what they are supposed to?

• What does the question below look like it If you said depression
would measure? you would be correct.
It looks like an item
Choose the item that best describes you: from a depression scale
(0) I do not feel sad.
– and it is!
(1) I feel sad.
(2) I am sad all the time and I can't snap out It has face validity.
of it.
(3) I am so sad or unhappy that I can't stand it. If someone said this
was from a parental
attachment scale, then
you could say that it
lacks face validity.
 Predictive Validity
Does the measure predict something that it logically should?
The GPA (Grade Point Average) in Mathematics
of Civil Engineering Students
What future measures do you expect that the GPA would be
correlated with?
If you said probability of passing the CE Board Examination,
you would be right, and the GPA is correlated with that! –

It has predictive validity

 Concurrent Validity
Does this measure correlate strongly with
something that it logically should right now?
Unlike predictive validity, concurrent validity
compares measures taken at the same time.
What current measures do you imagine that GPA
should correlate with?

If you said current teacher ratings you would be

correct.
GPA has some concurrent validity