Appraiser Patient Estimated State Effective State Repetition

Peter 1 Good Excellent 1

Peter 2 Very Bad Bad 1
Peter 3 Medium Medium 1
Peter 4 Very Bad Very Bad 1
Peter 5 Bad Medium 1
Peter 6 Good Good 1
Peter 7 Good Excellent 1
Peter 8 Medium Medium 1
Peter 9 Very Bad Bad 1
Peter 10 Medium Good 1
Peter 11 Very Bad Very Bad 1
Peter 12 Bad Medium 1
Peter 13 Excellent Excellent 1
Peter 14 Bad Bad 1
Peter 15 Good Good 1
Luis 1 Excellent Excellent 1
Luis 2 Bad Bad 1
Luis 3 Medium Medium 1
Luis 4 Very Bad Very Bad 1
Luis 5 Medium Medium 1
Luis 6 Good Good 1
Luis 7 Excellent Excellent 1
Luis 8 Medium Medium 1
Luis 9 Bad Bad 1
Luis 10 Excellent Good 1
Luis 11 Bad Very Bad 1
Luis 12 Medium Medium 1
Luis 13 Excellent Excellent 1
Luis 14 Bad Bad 1
Luis 15 Good Good 1
Tim 1 Excellent Excellent 1
Tim 2 Bad Bad 1
Tim 3 Medium Medium 1
Tim 4 Very Bad Very Bad 1
Tim 5 Medium Medium 1
Tim 6 Good Good 1
Tim 7 Excellent Excellent 1
Tim 8 Medium Medium 1
Tim 9 Bad Bad 1
Tim 10 Good Good 1
Tim 11 Very Bad Very Bad 1
Tim 12 Medium Medium 1
Tim 13 Excellent Excellent 1
Tim 14 Bad Bad 1
Tim 15 Good Good 1
Lea 1 Excellent Excellent 1
Lea 2 Bad Bad 1
Lea 3 Medium Medium 1
Lea 4 Very Bad Very Bad 1
Lea 5 Medium Medium 1
Lea 6 Good Good 1
Lea 7 Excellent Excellent 1
Lea 8 Medium Medium 1
Lea 9 Bad Bad 1
Lea 10 Good Good 1
Lea 11 Very Bad Very Bad 1
Lea 12 Medium Medium 1
Lea 13 Excellent Excellent 1
Lea 14 Bad Bad 1
Lea 15 Good Good 1
Zoe 1 Excellent Excellent 1
Zoe 2 Bad Bad 1
Zoe 3 Good Medium 1
Zoe 4 Very Bad Very Bad 1
Zoe 5 Medium Medium 1
Zoe 6 Good Good 1
Zoe 7 Excellent Excellent 1
Zoe 8 Medium Medium 1
Zoe 9 Bad Bad 1
Zoe 10 Good Good 1
Zoe 11 Very Bad Very Bad 1
Zoe 12 Medium Medium 1
Zoe 13 Excellent Excellent 1
Zoe 14 Bad Bad 1
Zoe 15 Good Good 1
Peter 1 Medium Excellent 2
Peter 2 Very Bad Bad 2
Peter 3 Medium Medium 2
Peter 4 Very Bad Very Bad 2
Peter 5 Bad Medium 2
Peter 6 Excellent Good 2
Peter 7 Good Excellent 2
Peter 8 Medium Medium 2
Peter 9 Very Bad Bad 2
Peter 10 Medium Good 2
Peter 11 Very Bad Very Bad 2
Peter 12 Bad Medium 2
Peter 13 Excellent Excellent 2
Peter 14 Bad Bad 2
Peter 15 Good Good 2
Luis 1 Good Excellent 2
Luis 2 Medium Bad 2
Luis 3 Medium Medium 2
Luis 4 Bad Very Bad 2
Luis 5 Medium Medium 2
Luis 6 Medium Good 2
Luis 7 Good Excellent 2
Luis 8 Medium Medium 2
Luis 9 Bad Bad 2
Luis 10 Good Good 2
Luis 11 Very Bad Very Bad 2
Luis 12 Medium Medium 2
Luis 13 Excellent Excellent 2
Luis 14 Bad Bad 2
Luis 15 Good Good 2
Tim 1 Excellent Excellent 2
Tim 2 Bad Bad 2
Tim 3 Medium Medium 2
Tim 4 Very Bad Very Bad 2
Tim 5 Medium Medium 2
Tim 6 Good Good 2
Tim 7 Excellent Excellent 2
Tim 8 Medium Medium 2
Tim 9 Bad Bad 2
Tim 10 Good Good 2
Tim 11 Very Bad Very Bad 2
Tim 12 Medium Medium 2
Tim 13 Excellent Excellent 2
Tim 14 Bad Bad 2
Tim 15 Good Good 2
Lea 1 Excellent Excellent 2
Lea 2 Bad Bad 2
Lea 3 Medium Medium 2
Lea 4 Very Bad Very Bad 2
Lea 5 Medium Medium 2
Lea 6 Good Good 2
Lea 7 Excellent Excellent 2
Lea 8 Medium Medium 2
Lea 9 Bad Bad 2
Lea 10 Good Good 2
Lea 11 Very Bad Very Bad 2
Lea 12 Medium Medium 2
Lea 13 Excellent Excellent 2
Lea 14 Bad Bad 2
Lea 15 Good Good 2
Zoe 1 Excellent Excellent 2
Zoe 2 Bad Bad 2
Zoe 3 Good Medium 2
Zoe 4 Very Bad Very Bad 2
Zoe 5 Medium Medium 2
Zoe 6 Good Good 2
Zoe 7 Excellent Excellent 2
Zoe 8 Medium Medium 2
Zoe 9 Bad Bad 2
Zoe 10 Good Good 2
Zoe 11 Very Bad Very Bad 2
Zoe 12 Medium Medium 2
Zoe 13 Excellent Excellent 2
Zoe 14 Bad Bad 2
Zoe 15 Good Good 2
XLSTAT 2015.1.01.12345 - Gage R&R / Attributes - on 2/2/2016 at 11:33:59 AM
Measurements: Workbook = demo-rra.xls / Sheet = data / Range = data!$C:$C / 150 rows and 1 column
Operators: Workbook = demo-rra.xls / Sheet = data / Range = data!$A:$A / 150 rows and 1 column
Parts: Workbook = demo-rra.xls / Sheet = data / Range = data!$B:$B / 150 rows and 1 column
Reference: Workbook = demo-rra.xls / Sheet = data / Range = data!$D:$D / 150 rows and 1 column
Confidence interval (%): 95

Within operator:

Within operator - Assessment agreement:

Operator Inspected Matched Percent Lower bound

Upper bound
Lea 15 15 100.00 81.90 100.00
Luis 15 8 53.33 26.59 78.73
Peter 15 13 86.67 59.54 98.34
Tim 15 15 100.00 81.90 100.00
Zoe 15 15 100.00 81.90 100.00

Assessment agreement
100

90

80
Percent

70

60

50

40

30

20

10

0
Lea Luis Peter Tim Zoe
Operators

Within operator - Fleiss' kappa:

Operator ResponseFleiss' kappa

Standard error Z p-value
Lea Bad 1.0000 0.2582 3.8730 < 0.0001
Excellent 1.0000 0.2582 3.8730 < 0.0001
 Good 1.0000 0.2582 3.8730 < 0.0001
Medium 1.0000 0.2582 3.8730 < 0.0001
Very Bad 1.0000 0.2582 3.8730 < 0.0001
Overall 1.0000 0.1313 7.6158 < 0.0001
Luis Bad 0.4410 0.2582 1.7080 0.0438
Excellent 0.2800 0.2582 1.0844 0.1391
Good 0.1667 0.2582 0.6455 0.2593
Medium 0.7000 0.2582 2.7111 0.0034
Very Bad -0.0714 0.2582 -0.2766 0.6090
Overall 0.3878 0.1392 2.7846 0.0027
Peter Bad 1.0000 0.2582 3.8730 < 0.0001
Excellent 0.6296 0.2582 2.4385 0.0074
Good 0.5833 0.2582 2.2592 0.0119
Medium 0.8137 0.2582 3.1513 0.0008
Very Bad 1.0000 0.2582 3.8730 < 0.0001
Overall 0.8300 0.1336 6.2140 < 0.0001
Tim Bad 1.0000 0.2582 3.8730 < 0.0001
Excellent 1.0000 0.2582 3.8730 < 0.0001
Good 1.0000 0.2582 3.8730 < 0.0001
Medium 1.0000 0.2582 3.8730 < 0.0001
Very Bad 1.0000 0.2582 3.8730 < 0.0001
Overall 1.0000 0.1313 7.6158 < 0.0001
Zoe Bad 1.0000 0.2582 3.8730 < 0.0001
Excellent 1.0000 0.2582 3.8730 < 0.0001
Good 1.0000 0.2582 3.8730 < 0.0001
Medium 1.0000 0.2582 3.8730 < 0.0001
Very Bad 1.0000 0.2582 3.8730 < 0.0001
Overall 1.0000 0.1313 7.6158 < 0.0001

Test interpretation:
H0: Kappa value is not greater than 0
Ha: Kappa value is greater than 0
If the computed p-value is lower than the significance level alpha=0.05, one should reject the null
hypothesis H0, and accept the alternative hypothesis Ha.

Within operator Cohen's kappa:

Operator ResponseCohen's kappa

Standard error Z p-value
Lea Bad 1.0000 0.2582 3.8730 < 0.0001
Excellent 1.0000 0.2582 3.8730 < 0.0001
Good 1.0000 0.2582 3.8730 < 0.0001
Medium 1.0000 0.2582 3.8730 < 0.0001
Very Bad 1.0000 0.2582 3.8730 < 0.0001
Overall 1.0000 0.1313 7.6158 < 0.0001
Luis Bad 0.4444 0.2537 1.7516 0.0399
Excellent 0.3284 0.1913 1.7165 0.0430
Good 0.1892 0.2360 0.8016 0.2114
Medium 0.7059 0.2468 2.8604 0.0021
Very Bad -0.0714 0.2582 -0.2766 0.6090
Overall 0.4034 0.1272 3.1718 0.0008
Peter Bad 1.0000 0.2582 3.8730 < 0.0001
Excellent 0.6341 0.2403 2.6390 0.0042
Good 0.5946 0.2360 2.5192 0.0059
Medium 0.8148 0.2537 3.2113 0.0007
Very Bad 1.0000 0.2582 3.8730 < 0.0001
Overall 0.8315 0.1296 6.4150 < 0.0001
Tim Bad 1.0000 0.2582 3.8730 < 0.0001
Excellent 1.0000 0.2582 3.8730 < 0.0001
Good 1.0000 0.2582 3.8730 < 0.0001
Medium 1.0000 0.2582 3.8730 < 0.0001
Very Bad 1.0000 0.2582 3.8730 < 0.0001
Overall 1.0000 0.1313 7.6158 < 0.0001
Zoe Bad 1.0000 0.2582 3.8730 < 0.0001
Excellent 1.0000 0.2582 3.8730 < 0.0001
Good 1.0000 0.2582 3.8730 < 0.0001
Medium 1.0000 0.2582 3.8730 < 0.0001
Very Bad 1.0000 0.2582 3.8730 < 0.0001
Overall 1.0000 0.1313 7.6158 < 0.0001

Test interpretation:
H0: Kappa value is not greater than 0
Ha: Kappa value is greater than 0
If the computed p-value is lower than the significance level alpha=0.05, one should reject the null
hypothesis H0, and accept the alternative hypothesis Ha.

Within operator - Kendall's coefficient of concordance:

Operator Coefficient Chi-square DF p-value

Lea 1.0000 28.0000 14 0.0142
Luis 0.5865 16.4231 14 0.2882
Peter 0.9869 27.6323 14 0.0159
Tim 1.0000 28.0000 14 0.0142
Zoe 1.0000 28.0000 14 0.0142

Test interpretation:
H0: Kendall's coefficient of concordance is equal to 0
Ha: Kendall's coefficient of concordance is greater than 0
If the computed p-value is lower than the significance level alpha=0.05, one should reject the null
hypothesis H0, and accept the alternative hypothesis Ha.
If the computed p-value is lower than the significance level alpha=0.05, one should reject the null
hypothesis H0, and accept the alternative hypothesis Ha.

Between operators:

Between operators - Assessment agreement:

Inspected Matched Percent Lower bound

Upper bound
15 4 26.67 7.79 55.10

Between operators - Fleiss' kappa:

ResponseFleiss' kappa
Standard error Z p-value
Bad 0.6069 0.0385 15.7688 < 0.0001
Excellent 0.6898 0.0385 17.9222 < 0.0001
Good 0.6028 0.0385 15.6600 < 0.0001
Medium 0.6496 0.0385 16.8784 < 0.0001
Very Bad 0.7041 0.0385 18.2923 < 0.0001
Overall 0.6467 0.0195 33.1476 < 0.0001

Test interpretation:
H0: Kappa value is not greater than 0
Ha: Kappa value is greater than 0
If the computed p-value is lower than the significance level alpha=0.05, one should reject the null
hypothesis H0, and accept the alternative hypothesis Ha.

Between operators Cohen's kappa:

Between operator Cohen's kappa can only be computed for two appraiser and single repetition

Between operators - Kendall's coefficient of concordance:

Coefficient Chi-square DF p-value

0.5748 80.4711 14 < 0.0001

Test interpretation:
H0: Kendall's coefficient of concordance is equal to 0
Ha: Kendall's coefficient of concordance is greater than 0
If the computed p-value is lower than the significance level alpha=0.05, one should reject the null
hypothesis H0, and accept the alternative hypothesis Ha.

Operator versus reference:

Operator versus reference - Assessment agreement:

Operator Inspected Matched Percent Lower bound

Upper bound
Lea 15 15 100.00 81.90 100.00
Luis 15 8 53.33 26.59 78.73
Peter 15 7 46.67 21.27 73.41
Tim 15 15 100.00 81.90 100.00
Zoe 15 14 93.33 68.05 99.83

Assessment agreement
100

90

80
Percent

70

60

50

40

30

20

10

0
Lea Luis Peter Tim Zoe
Operators

Operator versus reference - Fleiss' kappa:

Operator ResponseFleiss' kappa

Standard error Z p-value
Lea Bad 1.0000 0.1291 7.7460 < 0.0001
Excellent 1.0000 0.1291 7.7460 < 0.0001
Good 1.0000 0.1291 7.7460 < 0.0001
Medium 1.0000 0.1291 7.7460 < 0.0001
Very Bad 1.0000 0.1291 7.7460 < 0.0001
Overall 1.0000 0.0657 15.2317 < 0.0001
Luis Bad 0.6985 0.1291 5.4105 < 0.0001
Excellent 0.6184 0.1291 4.7899 < 0.0001
Good 0.6005 0.1291 4.6514 < 0.0001
Medium 0.8500 0.1291 6.5841 < 0.0001
Very Bad 0.6296 0.1291 4.8771 < 0.0001
Overall 0.6974 0.0678 10.2855 < 0.0001
Peter Bad 0.1667 0.1291 1.2910 0.0984
Excellent 0.3515 0.1291 2.7230 0.0032
Good 0.3605 0.1291 2.7924 0.0026
Medium 0.3796 0.1291 2.9403 0.0016
Very Bad 0.5833 0.1291 4.5185 < 0.0001
Overall 0.3697 0.0654 5.6561 < 0.0001
Tim Bad 1.0000 0.1291 7.7460 < 0.0001
Excellent 1.0000 0.1291 7.7460 < 0.0001
Good 1.0000 0.1291 7.7460 < 0.0001
Medium 1.0000 0.1291 7.7460 < 0.0001
Very Bad 1.0000 0.1291 7.7460 < 0.0001
Overall 1.0000 0.0657 15.2317 < 0.0001
Zoe Bad 1.0000 0.1291 7.7460 < 0.0001
Excellent 1.0000 0.1291 7.7460 < 0.0001
Good 0.8137 0.1291 6.3026 < 0.0001
Medium 0.8137 0.1291 6.3026 < 0.0001
Very Bad 1.0000 0.1291 7.7460 < 0.0001
Overall 0.9160 0.0655 13.9924 < 0.0001

Test interpretation:
H0: Kappa value is not greater than 0
Ha: Kappa value is greater than 0
If the computed p-value is lower than the significance level alpha=0.05, one should reject the null
hypothesis H0, and accept the alternative hypothesis Ha.

Operator versus reference Cohen's kappa:

Operator ResponseCohen's kappa

Standard error Z p-value
Lea Bad 1.0000 0.1826 5.4772 < 0.0001
Excellent 1.0000 0.1826 5.4772 < 0.0001
Good 1.0000 0.1826 5.4772 < 0.0001
Medium 1.0000 0.1826 5.4772 < 0.0001
Very Bad 1.0000 0.1826 5.4772 < 0.0001
Overall 1.0000 0.0928 10.7704 < 0.0001
Luis Bad 0.6991 0.1810 3.8622 < 0.0001
Excellent 0.6296 0.1662 3.7887 < 0.0001
Good 0.6032 0.1784 3.3815 0.0004
Medium 0.8529 0.1786 4.7762 < 0.0001
Very Bad 0.6341 0.1699 3.7321 < 0.0001
Overall 0.7010 0.0926 7.5710 < 0.0001
Peter Bad 0.1667 0.1826 0.9129 0.1807
Excellent 0.3651 0.1651 2.2118 0.0135
Good 0.3651 0.1784 2.0467 0.0203
Medium 0.3813 0.1810 2.1067 0.0176
 Very Bad 0.5946 0.1669 3.5626 0.0002
Overall 0.3766 0.0888 4.2421 < 0.0001
Tim Bad 1.0000 0.1826 5.4772 < 0.0001
Excellent 1.0000 0.1826 5.4772 < 0.0001
Good 1.0000 0.1826 5.4772 < 0.0001
Medium 1.0000 0.1826 5.4772 < 0.0001
Very Bad 1.0000 0.1826 5.4772 < 0.0001
Overall 1.0000 0.0928 10.7704 < 0.0001
Zoe Bad 1.0000 0.1826 5.4772 < 0.0001
Excellent 1.0000 0.1826 5.4772 < 0.0001
Good 0.8148 0.1794 4.5415 < 0.0001
Medium 0.8148 0.1794 4.5415 < 0.0001
Very Bad 1.0000 0.1826 5.4772 < 0.0001
Overall 0.9162 0.0917 9.9894 < 0.0001

Test interpretation:
H0: Kappa value is not greater than 0
Ha: Kappa value is greater than 0
If the computed p-value is lower than the significance level alpha=0.05, one should reject the null
hypothesis H0, and accept the alternative hypothesis Ha.

Operator versus reference - Kendall's correlation coefficient:

Operator Coefficient
Standard-error Z p-value
Lea 1.0000 0.1361 7.3135 < 0.0001
Luis 0.6074 0.1361 4.4284 < 0.0001
Peter 0.1078 0.1361 0.7569 0.4491
Tim 1.0000 0.1361 7.3135 < 0.0001
Zoe 0.9663 0.1361 7.0658 < 0.0001

Test interpretation:
H0: Kendall's correlation coefficient is equal to 0
Ha: Kendall's correlation coefficient is different from 0
If the computed p-value is lower than the significance level alpha=0.05, one should reject the null
hypothesis H0, and accept the alternative hypothesis Ha.

All versus reference:

All versus reference - Assessment agreement:

Inspected Matched Percent Lower bound

Upper bound
15 4 26.67 7.79 55.10
All versus reference - Fleiss' kappa:

ResponseFleiss' kappa
Standard error Z p-value
Bad 0.7730 0.0577 13.3893 < 0.0001
Excellent 0.7940 0.0577 13.7522 < 0.0001
Good 0.7549 0.0577 13.0758 < 0.0001
Medium 0.8087 0.0577 14.0062 < 0.0001
Very Bad 0.8426 0.0577 14.5941 < 0.0001
Overall 0.7966 0.0295 26.9915 < 0.0001

Test interpretation:
H0: Kappa value is not greater than 0
Ha: Kappa value is greater than 0
If the computed p-value is lower than the significance level alpha=0.05, one should reject the null
hypothesis H0, and accept the alternative hypothesis Ha.

All versus reference Cohen's kappa:

ResponseCohen's kappa
Standard error Z p-value
Bad 0.7731 0.0815 9.4854 < 0.0001
Excellent 0.7989 0.0787 10.1511 < 0.0001
Good 0.7566 0.0806 9.3849 < 0.0001
Medium 0.8098 0.0809 10.0134 < 0.0001
Very Bad 0.8457 0.0792 10.6817 < 0.0001
Overall 0.7988 0.0410 19.4628 < 0.0001

Test interpretation:
H0: Kappa value is not greater than 0
Ha: Kappa value is greater than 0
If the computed p-value is lower than the significance level alpha=0.05, one should reject the null
hypothesis H0, and accept the alternative hypothesis Ha.

All versus reference - Kendall's correlation coefficient:

Coefficient
Standard-error Z p-value
0.7363 0.0609 12.0828 < 0.0001

Test interpretation:
H0: Kendall's correlation coefficient is equal to 0
Ha: Kendall's correlation coefficient is different from 0
If the computed p-value is lower than the significance level alpha=0.05, one should reject the null
hypothesis H0, and accept the alternative hypothesis Ha.
s and 1 column