Types of Reliability

Types of Reliability
• Inter-Rater or Inter-Observer Reliability
Used to assess the degree to which different raters/observers give
consistent estimates of the same phenomenon.
• Test-Retest Reliability
Used to assess the consistency of a measure from one time to another.
• Parallel-Forms Reliability
Used to assess the consistency of the results of two tests constructed in
the same way from the same content domain.
• Internal Consistency Reliability
Used to assess the consistency of results across items within a test.
Reliability of Consistency of What?

• Observers or raters
• Tests over time
• Different versions of the same test
• A test at one point in time
Inter-Rater or Inter-Observer Reliability

Object or
phenomenon
Inter-Rater or Inter-Observer Reliability

Object or
phenomenon

Observer 1
Inter-Rater or Inter-Observer Reliability

Object or
phenomenon

Observer 1 Observer 2
Inter-Rater or Inter-Observer Reliability

Object or
phenomenon

?
=
Observer 1 Observer 2
Inter-Rater or Inter-Observer Reliability
• Are different observers consistent?
• Can establish this outside of your study in a pilot study.
• Can look at percent of agreement (especially with category ratings).
• Can use correlation (with continuous ratings).
Test-Retest Reliability

Time 1 Time 2
Test-Retest Reliability

Test = Test

Time 1 Time 2
Test-Retest Reliability

Stability over time

Test = Test

Time 1 Time 2
Test-Retest Reliability
• Measure instrument at two times for multiple persons.
• Compute correlation between the two measures.
• Assumes there is no change in the underlying trait between time 1
and time 2.
Parallel-Forms Reliability

Time 1 Time 2
Parallel-Forms Reliability

Form A

=
Form B

Time 1 Time 2
Parallel-Forms Reliability

Form A Stability across forms

=
Form B

Time 1 Time 2
Parallel-Forms Reliability
• Administer both forms to the same people.
• Get correlation between the two forms.
• Usually done in educational contexts where you need alternative
forms because of the frequency of retesting and where you can
sample from lots of equivalent questions.
Internal Consistency Reliability

Average inter-item correlation

Internal Consistency Reliability

Average Inter-Item Correlation

Test
Internal Consistency Reliability

Item 1 Average Inter-Item correlation

Item 2

Item 3
Test
Item 4

Item 5

Item 6
Internal Consistency Reliability

Item 1 Average inter-item correlation

Item 2 I1 I2 I3 I4 I5 I6
I1 1.00
Item 3 I2 .89 1.00
Test I3 .91 .92 1.00
Item 4 I4 .88 .93 .95 1.00
I5 .84 .86 .92 .85 1.00
I6 .88 .91 .95 .87 .85 1.00
Item 5

Item 6
Internal Consistency Reliability

Item 1 Average inter-item correlation

Item 2 I1 I2 I3 I4 I5 I6
I1 1.00
Item 3 I2 .89 1.00
Test I3 .91 .92 1.00
Item 4 I4 .88 .93 .95 1.00
I5 .84 .86 .92 .85 1.00
I6 .88 .91 .95 .87 .85 1.00
Item 5

Item 6
.90
Internal Consistency Reliability
Average item-total correlation
Internal Consistency Reliability
Average item-total correlation

Test
 Internal Consistency Reliability
Average item-total correlation
Item 1

Item 2

Item 3
Test
Item 4

Item 5

Item 6
 Internal Consistency Reliability
Average item-total correlation
Item 1
I1 I2 I3 I4 I5 I6

Item 2 I1 1.00
I2 .89 1.00
Item 3 I3 .91 .92 1.00
Test I4 .88 .93 .95 1.00
I5 .84 .86 .92 .85 1.00
Item 4 I6 .88 .91 .95 .87 .85 1.00
Total .84 .88 .86 .87 .83 .82 1.00
Item 5

Item 6
 Internal Consistency Reliability
Average item-total correlation
Item 1
I1 I2 I3 I4 I5 I6

Item 2 I1 1.00
I2 .89 1.00
Item 3 I3 .91 .92 1.00
Test I4 .88 .93 .95 1.00
I5 .84 .86 .92 .85 1.00
Item 4 I6 .88 .91 .95 .87 .85 1.00
Total .84 .88 .86 .87 .83 .82 1.00
Item 5

Item 6
.85
Internal Consistency Reliability
Split-half correlations
Internal Consistency Reliability
Split-half correlations

Test
 Internal Consistency Reliability
Split-half correlations
Item 1

Item 2

Item 3
Test
Item 4

Item 5

Item 6
 Internal Consistency Reliability
Split-half correlations
Item 1

Item 2 Item 1 Item 3 Item 4

Item 3
Test
Item 4

Item 5

Item 6
 Internal Consistency Reliability
Split-half correlations
Item 1

Item 2 Item 1 Item 3 Item 4

Item 3
Test
Item 4

Item 5
Item 2 Item 5 Item 6
Item 6
 Internal Consistency Reliability
Split-half correlations
Item 1

Item 2 Item 1 Item 3 Item 4

Item 3
Test
Item 4 .87
Item 5
Item 2 Item 5 Item 6
Item 6
Internal Consistency Reliability
Cronbach’s alpha ()
Internal Consistency Reliability
Cronbach’s alpha ()

Test
 Internal Consistency Reliability
Cronbach’s alpha ()
Item 1

Item 2

Item 3
Test
Item 4

Item 5

Item 6
 Internal Consistency Reliability
Cronbach’s alpha ()
Item 1
item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

Item 2 .87 .87 .87

item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3
Test
Item 4

Item 5

Item 6
 Internal Consistency Reliability
Cronbach’s alpha ()
Item 1
item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

Item 2 .87 .87 .87

item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3 SH1 .87

Test SH2 .85
Item 4 SH3 .91
SH4 .83
SH5 .86
Item 5 ...
SHn .85
Item 6
 Internal Consistency Reliability
Cronbach’s alpha ()
Item 1
item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

Item 2 .87 .87 .87

item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3 SH1 .87

Test SH2 .85
Item 4 SH3 .91
SH4 .83
SH5 .86
Item 5 ...
SHn .85
Item 6
 = .85
 Internal Consistency Reliability
Cronbach’s alpha ()
Item 1
item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

Item 2 .87 .87 .87

item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3 SH1 .87

Test SH2 .85
Item 4 SH3 .91 Like the average of
SH4 .83 all possible split-
SH5 .86
Item 5 ... half correlations
SHn .85
Item 6
 = .85
Internal Consistency Reliability - Summary

• Average inter-item correlation

• Average item-total correlation
• Split-half reliability
• Cronbach’s alpha ()