Types of Reliability

Reliability of Consistency of What?


Observers or raters

Tests over time

Different versions of the same test

A test at one point in time
Inter-Rater or Inter-Observer Reliability

Object or
phenomenon
Inter-Rater or Inter-Observer Reliability

Object or
phenomenon

Observer 1
Inter-Rater or Inter-Observer Reliability

Object or
phenomenon

Observer 1 Observer 2
Inter-Rater or Inter-Observer Reliability

Object or
phenomenon

?
=
Observer 1 Observer 2
Inter-Rater or Inter-Observer Reliability


Are different observers consistent?

Can establish this outside of your study
in a pilot study.

Can look at percent of agreement
(especially with category ratings).

Can use correlation (with continuous
ratings).
Test-Retest Reliability

Time 1 Time 2
Test-Retest Reliability

Test = Test

Time 1 Time 2
Test-Retest Reliability

Stability over time

Test = Test

Time 1 Time 2
 Test-Retest Reliability


Measure instrument at two times for
multiple persons.

Compute correlation between the two
measures.

Assumes there is no change in the
underlying trait between time 1 and time
2.
Parallel-Forms Reliability

Time 1 Time 2
Parallel-Forms Reliability

Form A
=
Form B

Time 1 Time 2
Parallel-Forms Reliability

Form A
Stability across forms

=
Form B

Time 1 Time 2
 Parallel-Forms Reliability


Administer both forms to the same
people.

Get correlation between the two forms.

Usually done in educational contexts
where you need alternative forms
because of the frequency of retesting
and where you can sample from lots of
equivalent questions.
 The Correlation Coefficient


Credited to Karl Pearson (1896)

Measures the degree of linear
association between two variables.

Ranges from -1.0 to 1.0

Sign refers to direction
– Negative: As X increases Y decreases
– Positive: As X increases Y increases
 One Formula


Symbolized by r

Covariance of X and Y Divided by the
Product of the SDs of X and Y.

covXY
r
s X sY
 Calculation of r for
Payroll (X) and Winning Percentage (Y)

 covXY = 1.13
 sX = 34.23
 sY = .07

cov XY 1.13 1.13

r     .47
s X sY ( 34.23)( 0.07) 2.40
 Calculation of r for
Payroll (X) and Making Post-Season (Y)


Y coded so that 1=Playoffs 0=No
 covXY = 8.24
 sX = 34.23
 sY = .45

cov XY 8.24 8.24

r     .53
s X sY ( 34.23)( 0.45) 15.42
 Examples of Correlations
Source: Meyer et al. (2001)

Associations r
Test Anxiety and Grades -.17
SAT and Grades in College .20
GRE Quant. and Graduate School GPA .22
Quality of Marital Relationships and Quality of .22
Parent-Child Relationships
Alcohol and Aggressive Behavior .23
Height and Weight .44
Gender and Height .67
 Commonly Used Rule of Thumb


+/- .10 is Small

+/- .30 is Medium

+/- .50 is Large

Use these with care. This guidelines
only provide a loose framework for
thinking about the size of correlations

Sources: Cohen (1988) and Kline
(2004)
 r=0
4
3
2
1
true

0
-4 -3 -2 -1 -1 0 1 2 3 4
-2
-3
-4
observed
 r=.10
4
3
2
1
true

0
-4 -3 -2 -1 -1 0 1 2 3 4
-2
-3
-4
observed
 r=.20
4
3
2
1
true

0
-4 -3 -2 -1 -1 0 1 2 3 4
-2
-3
-4
observed
 r=.30
4
3
2
1
true

0
-4 -3 -2 -1 -1 0 1 2 3 4
-2
-3
-4
observed
 r=.40
4
3
2
1
true

0
-4 -3 -2 -1 -1 0 1 2 3 4
-2
-3
-4
observed
 r=.50
4
3
2
1
true

0
-4 -3 -2 -1 -1 0 1 2 3 4
-2
-3
-4
observed
 r=.60
4
3
2
1
true

0
-4 -3 -2 -1 -1 0 1 2 3 4
-2
-3
-4
observed
 r=.70
4
3
2
1
true

0
-4 -3 -2 -1 -1 0 1 2 3 4
-2
-3
-4
observed
 r=.80
4
3
2
1
true

0
-4 -3 -2 -1 -1 0 1 2 3 4
-2
-3
-4
observed
 r=.90
4
3
2
1
true

0
-4 -3 -2 -1 -1 0 1 2 3 4
-2
-3
-4
observed
 r=1.0
4
3
2
1
true

0
-4 -3 -2 -1 -1 0 1 2 3 4
-2
-3
-4
observed
Internal Consistency Reliability

Average inter-item correlation

Internal Consistency Reliability

Average Inter-Item Correlation

Test
Internal Consistency Reliability

Item 1 Average Inter-Item correlation

Item 2

Item 3
Test
Item 4

Item 5

Item 6
Internal Consistency Reliability

Item 1 Average inter-item correlation

Item 2 I1 I2 I3 I4 I5 I6

I1 1.00
Item 3 I2 .89 1.00
Test I3 .91 .92 1.00
Item 4 I4 .88 .93 .95 1.00
.84 .86 .92 .85 1.00
I5
.88 .91 .95 .87 .85 1.00
Item 5 I6

Item 6
Internal Consistency Reliability

Item 1 Average inter-item correlation

Item 2 I1 I2 I3 I4 I5 I6

I1 1.00
Item 3 I2 .89 1.00
Test I3 .91 .92 1.00
Item 4 I4 .88 .93 .95 1.00
.84 .86 .92 .85 1.00
I5
.88 .91 .95 .87 .85 1.00
Item 5 I6

Item 6
.90
Internal Consistency
Reliability
Average item-total correlation
Internal Consistency
Reliability
Average item-total correlation

Test
 Internal Consistency
Reliability
Average item-total correlation
Item 1

Item 2

Item 3
Test
Item 4

Item 5

Item 6
 Internal Consistency
Reliability
Average item-total correlation
Item 1
I1 I2 I3 I4 I5 I6

Item 2 I1 1.00
I2 .89 1.00
Item 3 I3 .91 .92 1.00
Test I4 .88 .93 .95 1.00
.84 .86 .92 .85 1.00
Item 4 I5
.88 .91 .95 .87 .85 1.00
I6 .84 .88 .86 .87 .83 .82 1.00
Item 5 Total

Item 6
 Internal Consistency
Reliability
Average item-total correlation
Item 1
I1 I2 I3 I4 I5 I6

Item 2 I1 1.00
I2 .89 1.00
Item 3 I3 .91 .92 1.00
Test I4 .88 .93 .95 1.00
.84 .86 .92 .85 1.00
Item 4 I5
.88 .91 .95 .87 .85 1.00
I6 .84 .88 .86 .87 .83 .82 1.00
Item 5 Total

Item 6
.85
Internal Consistency
Reliability
Split-half correlations
Internal Consistency
Reliability
Split-half correlations

Test
 Internal Consistency
Reliability
Split-half correlations
Item 1

Item 2

Item 3
Test
Item 4

Item 5

Item 6
 Internal Consistency
Reliability
Split-half correlations
Item 1

Item 2 Item 1 Item 3 Item 4

Item 3
Test
Item 4

Item 5

Item 6
 Internal Consistency
Reliability
Split-half correlations
Item 1

Item 2 Item 1 Item 3 Item 4

Item 3
Test
Item 4

Item 5
Item 2 Item 5 Item 6
Item 6
 Internal Consistency
Reliability
Split-half correlations
Item 1

Item 2 Item 1 Item 3 Item 4

Item 3
Test
Item 4 .87
Item 5
Item 2 Item 5 Item 6
Item 6
Internal Consistency
Reliability
Cronbach’s alpha ()
Internal Consistency
Reliability
Cronbach’s alpha ()

Test
 Internal Consistency
Reliability
Cronbach’s alpha ()
Item 1

Item 2

Item 3
Test
Item 4

Item 5

Item 6
 Internal Consistency
Reliability
Cronbach’s alpha ()
Item 1 item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

Item 2 .87 .87 .87

item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3
Test
Item 4

Item 5

Item 6
 Internal Consistency
Reliability
Cronbach’s alpha ()
Item 1 item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

Item 2 .87 .87 .87

item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3 SH1 .87

Test SH2 .85
Item 4 SH3 .91
SH4 .83
Item 5 SH5 .86
...
Item 6 SHn .85
 Internal Consistency
Reliability
Cronbach’s alpha ()
Item 1 item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

Item 2 .87 .87 .87

item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3 SH1 .87

Test SH2 .85
Item 4 SH3 .91
SH4 .83
Item 5 SH5 .86
...
Item 6 SHn .85
 = .85
 Internal Consistency
Reliability
Cronbach’s alpha ()
Item 1 item 1 item 3 item 4 item 1 item 3 item 4 item 1 item 3 item 4

Item 2 .87 .87 .87

item 2 item 5 item 6 item 2 item 5 item 6 item 2 item 5 item 6

Item 3 SH1 .87

Test SH2 .85
Item 4 SH3 .91 Like the average
SH4 .83 of all possible
Item 5 SH5 .86 split-half
... correlations
Item 6 SHn .85
 = .85
Internal Consistency Reliability - Summary


Average inter-item correlation

Average item-total correlation

Split-half reliability

Cronbach’s alpha ()