[ research report ]

Terese L. Chmielewski, PT, PhD, SCS1 • Michael J. Hodges, PT, MHS2 • MaryBeth Horodyski, EdD, ATC, LAT3
Mark D. Bishop, PT, PhD, CSCS1 • Bryan P. Conrad, M Eng4 • Susan M. Tillman, PT, CSCS, SCS5

Investigation of Clinician Agreement in

Evaluating Movement Quality During
Unilateral Lower Extremity Functional Tasks:
A Comparison of 2 Rating Methods
Downloaded from www.jospt.org at on November 7, 2015. For personal use only. No other uses without permission.

B
iomechanical analysis has been used to identify faulty reduced peak coronal plane knee motion
lower extremity movement patterns for the purposes of during a stepping task has been reported
to accompany an improvement in patel-
preventing and improving the rehabilitation of knee injuries.
lofemoral pain.11 The knee is part of a
Copyright © 2007 Journal of Orthopaedic & Sports Physical Therapy®. All rights reserved.

Coronal plane knee movement is a key focus area. For kinetic chain and, as such, movement
example, increased coronal plane motion at the knee (peak knee at joints above and below can influence
valgus angle) when landing from a jump is associated with higher knee position. The influence of the hip
risk for anterior cruciate ligament injury in females.9 In addition, appears to be particularly important to
coronal plane knee position. Hip adduc-
t Study Design: Nonexperimental. t Results: Interrater and intrarater percent tion accompanies a valgus knee position
during a variety of functional tasks, 9,11,18
t Objectives: To determine interrater and agreement were higher using the overall method.
intrarater agreement for 2 methods of evaluating Interrater weighted kappa coefficients were and hip internal rotation is associated
movement quality during 2 lower extremity func- similar between rating methods (overall method, with valgus knee moment during a side-
Journal of Orthopaedic & Sports Physical Therapy®

tional tasks, and to descriptively compare levels of
agreement between the 2 methods.
t Background: Clinicians typically use
observational analysis to evaluate movement
quality during functional tasks, but the extent of
agreement is unknown.
t Methods and Measures: Twenty-five
uninjured subjects performed 3 trials of unilateral
squat and lateral step-down tasks. Three clinicians
evaluated the trunk, pelvis, and hips for coronal
plane and transverse plane movement deviations.
Two rating methods were used: assessment of the
entire movement (“overall method”) and rating
each segment individually (“specific method”).
Movement deviation severity was rated using basic
clinical guidelines and ratings were repeated from
videotape. Percent agreement and weighted kappa
coefficients were calculated between rater pairs
and rating sessions. Generalized kappa coefficients
were calculated across raters.
0-0.55; specific method, 0.23-0.53). Intrarater
weighted kappa coefficients were higher for the
specific method (0.38-0.68) compared to the
overall method (0.13-0.50). Generalized kappa
coefficients were also higher for specific method
compared to the overall method (unilateral squat,
0.19 and 0.01, respectively; lateral step-down,
0.22 and 0.18, respectively) and 95% confidence
intervals remained above zero.
t Conclusions: Rating movement at body
segments appears to result in agreement among
raters that is better than chance. Neither rating
method produced high agreement, indicating a
need to develop more explicit criteria for rating
movement deviation severity. J Orthop Sports
Phys Ther 2007;37(3):122-129. doi:10.2519/
jospt.2007.2457
t Key Words: functional testing, hip, knee,
movement analysis, neuromuscular, reliability
stepping task.12 Trunk position may ex-
ert an additional, indirect influence on
coronal plane knee position by altering
the location of the center of gravity rela-
tive to the lower extremity and by affect-
ing the length-tension relationship of
muscles that control the hip.17 For this
reason, several studies have included the
trunk in the biomechanical analysis of
lower extremity tasks.6,18
Clinicians rarely have access to the
equipment and expertise needed for bio-
mechanical analysis of movement and,
instead, rely on visual analysis to make
judgments about movement quality.
Most clinicians receive basic instruction
in visually analyzing movement patterns
for common functional tasks (eg, gait)
during their formal education. Instruc-

Assistant Professor, Department of Physical Therapy, University of Florida, Gainesville, FL. 2 Staff Physical Therapist, UF&Shands, Gainesville, FL. 3 Associate Professor, Director
1 

of Research, Department of Orthopaedics and Rehabilitation, University of Florida, Gainesville, FL. 4 Biomedical Engineer, Department of Orthopaedics and Rehabilitation,
University of Florida, Gainesville, FL. 5 Clinical Coordinator, Shands Rehab at the Orthopaedics and Sports Medicine Institute, Gainesville, FL. This work was supported by a grant
from the University of Florida Research Opportunity Fund. The University of Florida Institutional Review Board approved the protocol for this study. Address correspondence to
Terese L. Chmielewski, University of Florida, Department of Physical Therapy, PO Box 100154, HSC, Gainesville, FL 32610. E-mail: tchm@ufl.edu

122 | march 2007 | volume 37 | number 3 | journal of orthopaedic & sports physical therapy
 tion is typically focused on identifying
the presence of specific movement de-
viations (eg, increased knee flexion dur-
ing midstance) and final assessments
are often based on the identification of
multiple possible deviations. In con-
trast, instruction does not usually in-
clude guidelines for rating the severity
of movement deviation, nor does it en-
compass the wide variety of functional
tasks that may be observed. Without
structure for visually analyzing move-
ment it is possible that clinicians could
Downloaded from www.jospt.org at on November 7, 2015. For personal use only. No other uses without permission.
make dissimilar conclusions about the
same patient and, ultimately, apply dif-
ferent treatment plans.
Recently, greater emphasis has been
placed on visually analyzing movement
patterns during functional tasks to iden-
Copyright © 2007 Journal of Orthopaedic & Sports Physical Therapy®. All rights reserved.
tify candidates for knee injury prevention
programs or neuromuscular control inter-
ventions during knee rehabilitation.3,10,17
The unilateral squat1,10,17 and lateral step-
down tasks14 have been used because they
can elicit compensations that lead to knee
valgus.10,17 Understanding the extent of
clinician agreement in rating movement
quality for these functional tasks is a
Journal of Orthopaedic & Sports Physical Therapy®
necessary step toward developing stan-
dardized visual analysis methods. In this
study, we compared 2 rating methods.
The first method involved an assessment
of the entire movement pattern (“overall
method”), for which quality of move-
ment was classified into 1 of 3 categories
(“good,” “fair,” or “poor”). The second rat-
ing method entailed identifying specific
movement deviations and assigning a
severity rating at individual body seg-
ments (“specific method”). The purpose
FIGURE 1. A subject performing the unilateral squat task.
journal of orthopaedic & sports physical therapy | volume 37 | number 3 | march 2007 |
of this study was to (1) determine inter-
rater and intrarater agreement for the 2
methods of rating quality of movement
and (2) descriptively compare levels of
agreement between the 2 rating methods.
We hypothesized that the specific method
would result in higher interrater and in-
trarater agreement.
METHODS
Subjects
T
wenty-five subjects between
the ages of 18 and 25 years partici-
pated in this experiment (7 males,
18 females; mean 6 SD age, 22.4 6 1.3
years; mean 6 SD height, 168.9 6 9.1
cm; mean 6 SD body mass, 66.4 6 13.6
kg). All subjects were free from pain in
their low back or lower extremities at the
time of testing. Subjects were excluded
from participation if they reported (1) a
history of significant low back or lower
extremity injury that required physician
care, (2) balance or vestibular problems,
or (3) an incidence of concussion in the
month prior to testing. The test limb
was the limb used for stance when kick-
ing a ball. Each subject provided written
informed consent. The protocol for this
study was approved by the University of
Florida Institutional Review Board.
Testing Protocol
The functional tasks chosen for this study
were the unilateral squat and the lateral
step-down, which are shown in Figures 1
and 2. These tasks were chosen because
they are used clinically, are easy to admin-
ister, are performed unilaterally, and can
induce coronal plane and transverse plane
changes at the pelvis and hip if weakness
is present.7,10,17 In addition, initial pilot
testing of other functional tasks dem-
onstrated that rater vision was obscured
during reaching tests and that tasks per-
formed in bilateral stance were not suffi-
ciently demanding to uninjured subjects
without the inclusion of external weight.
All subjects reviewed a written de-
scription of the functional tasks and were
given standardized training that included
verbal instruction in the task and a visual
demonstration. Subjects were allowed 3
to 5 practice trials of each task, during
which they were given verbal cues to re-
mind them to maintain neutral trunk,
pelvis, and hip position in the coronal and
transverse planes. Subjects also received
verbal feedback if they were performing
the task incorrectly. Electromagnetic
motion-tracking sensors (Fastrak; Pol-
hemus, Colchester, VT) were then placed
on the foot, shank, and thigh of the test
lower extremity and on the pelvis. These
sensors were used to standardize the
knee flexion angle during the unilateral
squat task. The order of functional tasks
was randomized among subjects using a
random-number generator. Three trials
of each functional task were collected to
obtain a general representation of sub-
ject performance. A trial was repeated if
a subject experienced a loss of balance or
failed to meet the performance require-
ments of the task (eg, achieving 60° of
knee flexion during the unilateral squat
or touching the contralateral heel to the
floor during the lateral step-down). Cli-
nician raters were positioned 3 m (9 ft)
away from the subject, next to a digital
camcorder that was used to record all tri-
als for retrospective review.
Unilateral Squat  The starting position
for the unilateral squat was standing on
the test leg with the hip and knee in a
neutral anatomical position. The trunk
was upright, without rotation or lateral
flexion, and the contralateral leg was po-
sitioned with the hip in neutral and the
knee in approximately 90° of flexion. Sub-
FIGURE 2. A subject performing the lateral step-down task.
123
 [ research report ]
were considered a movement pattern de-
Scoring Used for the Specific Method of
TABLE 1 viation. Pelvic drop in the coronal plane
Evaluating Movement Quality*
was also considered a movement pattern
Symbol Point Value Criteria deviation for the lateral step-down task.
0 0 No deviation from neutral alignment Furthermore, oscillatory movements of
| 5 A small-magnitude or barely observable movement out of a neutral position and/or low these segments (repetitive movement
frequency of segment oscillation away from and back toward neutral) were
√ 10 A moderate-magnitude or marked movement out of a neutral position and/or moderate considered representative of a movement
frequency of segment oscillation
pattern deviation, with greater frequency
X 15 Excessive or severe magnitude of movement out of a neutral position and/or high
frequency of segment oscillation
and magnitude of oscillation representing
a more severe deviation. Recent biome-
*A symbol was used to express the rater’s assessment of adherence to a neutral position at each segment
(trunk, hips, and pelvis) during the functional task performance. Each symbol corresponded to a point chanical studies support lower extremity
value that increased with increasing severity of deviation. oscillations as a movement deviation.5,13
Downloaded from www.jospt.org at on November 7, 2015. For personal use only. No other uses without permission.

Overall Method Scoring System  The

jects moved at a self-selected pace into a
squat position, were given a verbal signal
when the test knee reached 60° of flexion,
and then returned to the starting position.
Trials were repeated if the subject did not
Copyright © 2007 Journal of Orthopaedic & Sports Physical Therapy®. All rights reserved.
Movement Assessment
Clinician Raters  Three clinicians (2
physical therapists and 1 athletic trainer
with a mean 6 SD of 14 6 6 years of clini-
cal experience) rated movement patterns
overall method of evaluating move-
ment quality required categorization
of the movement pattern as good, fair,
or poor. In keeping with current clini-
cal practice, raters were only given ba-

reach 60° of knee flexion or if balance
was lost, resulting in placement of the
contralateral foot on the floor. A 60° knee
flexion angle was chosen for this task be-
cause it represents an angle achieved dur-
ing dynamic tasks like cutting,13 and pilot
testing revealed that some subjects were
unable to achieve 90° of knee flexion.
Lateral Step-Down  For the lateral
Journal of Orthopaedic & Sports Physical Therapy®
for each functional task. Methods for rat-
ing movement quality were developed by
the raters in conjunction with another
physical therapist with 10 years of clini-
cal experience and were meant to reflect
current clinical practice. After initial de-
velopment, the raters used the scoring
methods to assess movement quality for
11 uninjured subjects and subsequently
sic guidelines for scoring the severity
of movement deviation. Raters were
instructed to consider the regions of
the body discussed above in scoring the
movement. A good score represented no
deviation in the movement pattern; a
fair score was given if movement devia-
tions were barely observable; and a poor
score indicated marked deviations in

step-down, subjects stood on the test leg,
which was positioned on the edge of an
adjustable step (Reebok Step System;
Reebok International, Canton, MA), with
the hip and knee in a neutral anatomical
position. The trunk was upright without
rotation or lateral flexion, the iliac crests
were level, and the contralateral leg was
unsupported, with the hip in a slightly
flexed position and the knee extended
(Figure 2). Subjects lowered themselves
at a self-selected pace until the contralat-
eral heel contacted the ground and then
returned to the starting position. Step
height ranged between 15.24 and 25.4 cm
and was determined by the height of the
subject (subjects under 163 cm in height
used a 15.24-cm step; those between 163
and 180 cm in height used a 20.32-cm
step; subjects over 180 cm in height used
a 25.4-cm step). Trials were repeated if
the subject failed to touch the contralat-
eral heel to the ground or stepped off the
adjustable step due to loss of balance.
met to discuss and resolve any confusion
regarding the rating guidelines. For this
study, the raters assessed movement pat-
terns at 2 different times: the first scoring
occurred while the subject performed the
functional task and the second scoring oc-
curred when the raters viewed the video-
taped performance. Testing occurred over
a span of 6 weeks and scoring of the video-
tape occurred an average (6SD) of 10.0 6
1.5 weeks after the completion of testing.
During the first scoring, raters refrained
from making comments related to the
subject performance and were required to
complete scoring within 30 seconds after
the subject completed the functional task.
While reviewing the videotaped perfor-
mance, raters were asked to limit them-
selves to 1 opportunity for rewinding the
videotape.
Movement Deviations  For both func-
tional tasks, trunk and hip movement
away from the neutral starting position
in the coronal and transverse planes
the movement pattern. Raters were not
given instruction in resolving situations
where a more severe deviation might be
present at 1 segment compared to the
other segments.
Specific Method Scoring System  The
specific method involved scoring each
segment (trunk, pelvis, and hip) individu-
ally, using a set of symbols to indicate the
severity of deviation from a neutral posi-
tion (Table 1). Each symbol corresponded
to a numerical equivalent that was set
in multiples of 5, with higher numbers
indicating more severe deviation from a
neutral position. Similar to the overall
method, raters were only given basic in-
struction for determining “mild,” “mod-
erate” and “severe” movement deviations
(Table 1), so that the main difference be-
tween the specific and overall methods
of evaluating movement quality was that
the specific method required the scoring
of individual body segments. In this way,
the specific method reflected movement

124 | march 2007 | volume 37 | number 3 | journal of orthopaedic & sports physical therapy
 served values)/∑(weighted expected
Frequency Distribution of Ratings for Each
values)].
TABLE 2 Functional Task Using the Overall Method of
The weighting was configured such
Evaluating Lower Extremity Movement Quality
that larger discrepancies in agreement
Poor Fair Good Total were assigned a greater penalty. We
Unilateral squat chose this analysis because, while an
Rater 1 1 14 10 25 unweighted kappa treats all disagree-
Rater 2 2 18 5 25 ment as equivalent, weighted kappa
Rater 3 ... 10 12 22 can penalize larger discrepancies in
Total* 3 (4.2) 42 (58.3) 27 (37.5) 72 (100) agreement16 that would likely result in
Lateral step-down different treatment plans. Figure 3 pro-
Rater 1 ... 17 8 25 vides an example of the calculation of
Rater 2 7 15 3 25 the weighted kappa statistic. Interrater
Downloaded from www.jospt.org at on November 7, 2015. For personal use only. No other uses without permission.

Rater 3 1 18 3 22 agreement was determined between

Total* 8 (11.2) 50 (69.4) 14 (19.4) 72 (100) each combination of raters, creating 3
*The percentage of all ratings is included in parentheses. comparisons. Intrarater agreement was
calculated for each rater by comparing
scoring from the actual trials to scoring
deviations at all segments, whereas in the the specific method was used in statis- from videotape.
Copyright © 2007 Journal of Orthopaedic & Sports Physical Therapy®. All rights reserved.

overall method there was potential for
more severe deviations at 1 segment to be
minimized in the overall score. Symbols
were used to improve scoring efficiency
and a trial score was computed by sum-
ming all deviations. Scoring movement
deviations in multiples of 5 was used to
assist in tallying the trial score.
Journal of Orthopaedic & Sports Physical Therapy®
tical analyses, as opposed to individual
segment scores, because the trial score
reflected the entire movement pattern as
rated with the overall method.
Data for each functional task were
analyzed separately. A general indication
of agreement was obtained for both scor-
ing methods by calculating the percent
agreement (number of exact agreements/
Additionally, a generalized kappa
coefficient was calculated. Generalized
kappa coefficients can be used to com-
pare multiple raters.4 In contrast to the
weighted kappa statistic that compares
observed frequencies of agreement and
those expected by chance, the general-
ized kappa represents a comparison of
the proportion of possible rater agree-

Data Management and
Analysis
The median score from the 3 trials was
computed for each rater and used in all
statistical analyses. The trial score from
Statistical number of possible scores). In addition, ments to the proportion of classifica-
a symmetrically weighted kappa statistic tions in each category.
was used to determine rater agreement
greater than that expected by chance2: RESULTS
weighted kappa = 1 – [∑(weighted ob-

T
he left leg was the test leg for
24 of 25 subjects. Raters 1 and 2
Frequency Distribution of Ratings for Each evaluated movement patterns for all
TABLE 3 Functional Task Using the Specific Method of 25 subjects, whereas rater 3 only evalu-
Evaluating Lower Extremity Movement Quality* ated 22 of 25 subjects due to scheduling
constraints.
0 5 10 15 20 25 30 Total
Ratings for both functional tasks were
Unilateral squat
clustered, regardless of method used for
Rater 1 2 8 9 6 ... ... ... 25
evaluating movement (Tables 2 and 3).
Rater 2 ... 6 8 8 2 1 ... 25
The majority of the overall method rat-
Rater 3 1 9 5 7 ... ... ... 22
ings fell into the fair and good categories
Total† 3 (4.2) 23 (31.9) 22 (30.5) 21 (29.2) 2 (2.8) 1 (1.4) ... 72 (100)
(unilateral squat, ~96%; lateral step-
Lateral step-down
down, ~89%). Similarly, using the spe-
Rater 1 ... 8 10 7 ... ... ... 25
cific method, almost 92% of the ratings
Rater 2 ... 2 5 4 7 5 2 25
for the unilateral squat were in the 5- to
Rater 3 ... 3 10 7 2 ... ... 22
15-point range, and 90.3% of the ratings
Total† ... 13 (18.1) 25 (34.7) 18 (25) 9 (12.5) 5 (6.9) 2 (2.8) 72 (100)
for the lateral step-down were in the 5-
*Based on ratings of live performance.
†
The percentage of all ratings is included in parentheses.
to 20-point range.
Although percent agreement between

journal of orthopaedic & sports physical therapy | volume 37 | number 3 | march 2007 | 125
 [ research report ]
Interrater Agreement for the Overall and Specific
TABLE 4
Methods of Evaluating Movement Quality*

Overall Method Specific Method

Rater 1 Versus 2 Rater 2 Versus 3 Rater 1 Versus 3 Rater 1 Versus 2 Rater 2 Versus 3 Rater 1 Versus 3
Percent agreement
Unilateral squat 60 64 41 36 41 32
Lateral step-down 56 82 55 20 50 27
Weighted kappa
Unilateral squat 0.37 (0.27 to 0.47) 0.18 (0.05 to 0.26) 0 (–0.11 to 0.11) 0.42 (0.34 to 0.60) 0.29(0.02 to 0.59) 0.41 (0.16 to 0.66)
Lateral step-down 0.32 (0.21 to 0.43) 0.55 (0.49 to 0.61) 0.21 (0.12 to 0.31) 0.23 (0.07 to 0.37) 0.53 (0.41 to 0.69) 0.31 (0.21 to 0.55)
Generalized kappa
Downloaded from www.jospt.org at on November 7, 2015. For personal use only. No other uses without permission.

Unilateral squat 0.01 (–0.27 to 0.25) 0.18 (0.04 to 0.32)

Lateral step-down 0.19 (–0.15 to 0.53) 0.22 (0.07 to 0.36)
* Based on ratings of live performance. Percentage agreement and weighted kappa coefficients are given for each combination of raters. The generalized kappa
coefficient, which summarizes agreement across raters, is given along with 95% confidence intervals in parentheses.

raters was higher using the overall meth- DISCUSSION beyond that expected by chance) was
Copyright © 2007 Journal of Orthopaedic & Sports Physical Therapy®. All rights reserved.

od, and values from the weighted and only 0.18. In addition, 95% confidence

generalized kappa statistics tended to be
higher for the specific method (Table 4).
Percent agreement between rater pairs
ranged from 41% to 82% for the overall
method and 20% to 50% for the specific
method. Weighted kappa coefficients for
rater pairs ranged from 0.00 to 0.55 for
the overall method compared to 0.23 to
Journal of Orthopaedic & Sports Physical Therapy®
T
he purpose of this study was
to determine interrater and intra-
rater agreement for 2 methods of
rating quality of movement during uni-
lateral lower extremity functional tasks
and to compare levels of rater agreement
between the 2 rating methods. We hy-
pothesized that evaluating movement
intervals for the generalized kappa co-
efficients of the overall method include
zero, which indicates that no agreement
beyond that expected by chance is a po-
tential result.
Requiring clinicians to rate specif-
ic body segments separately (specific
method) had the opposite effect. Agree-

0.53 for the specific method. General-
ized kappa coefficients for the overall
method were 0.01 and 0.19 (unilateral
squat and lateral step-down, respective-
ly), and 0.18 and 0.22 (unilateral squat
and lateral step-down, respectively) for
the specific method. The 95% confidence
intervals for generalized kappa coeffi-
cients included zero for both functional
tasks using the overall method, whereas
they did not for the specific method.
Results for intrarater agreement fol-
lowed a similar trend. Percent agree-
ment was higher for the overall method
and weighted kappa coefficients were
more favorable for the specific method
(Table 5). The intrarater percent agree-
ment ranged from 56% to 76% for the
overall method and 32% to 60% for the
specific method. In contrast, weighted
kappa coefficients for intrarater agree-
ment ranged from 0.13 to 0.50 for the
overall method and 0.38 to 0.68 for the
specific method.
at individual body segments (specific
method) would result in better interrater
and intrarater agreement than a more
global assessment of movement (overall
method). Our data partially support our
hypothesis. The percent agreement was
higher between and within raters us-
ing the overall method compared to the
specific method. In contrast, the values
generated by kappa statistics, which de-
termine agreement beyond that expected
by chance alone, tended to be higher for
the specific method.
Based on our results, the clinical
practice of rating movement patterns
into good, fair, or poor categories (over-
all method) without explicit guidelines
leads to clinician agreement a little over
half the time. Results from the kappa
statistics, however, are less favorable. For
example, raters 2 and 3 had 64% agree-
ment in rating movement patterns dur-
ing the unilateral squat, but the weighted
kappa coefficient (indicating agreement
ment for rater pairs never exceeded 50%
with the specific method, which is lower
than the overall method. This result is not
unexpected because the greater number
of scoring options in the specific method
compared to the overall method in-
creases the probability of disagreement.
On the other hand, point estimates
for weighted kappa statistics reached
similar magnitude between the 2 rat-
ing methods, and the lowest weighted
kappa coefficient of the specific meth-
od had a higher value than the overall
method. More importantly, 95% confi-
dence intervals for generalized kappa co-
efficients of the specific method did not
contain zero. This indicates, at the very
minimum, that rater agreement using
the specific method will almost always
produce agreement better than expected
by chance alone.
Results for intrarater agreement fol-
lowed the trends of interrater agreement.
That is, the overall method produced

126 | march 2007 | volume 37 | number 3 | journal of orthopaedic & sports physical therapy
 Intrarater Agreement for the Overall and Specific
TABLE 5
Methods of Evaluating Movement Quality*

Overall Method Specific Method

Rater 1 Rater 2 Rater 3 Rater 1 Rater 2 Rater 3
Percent agreement
Unilateral squat 64 60 59 40 48 32
Lateral step-down 56 76 68 40 60 50
Weighted kappa
Unilateral squat 0.42 (0.12 to 0.72) 0.29 (–0.06 to 0.64) 0.18 (–0.25 to 0.62) 0.53 (0.31 to 0.75) 0.38 (0.11 to 0.65) 0.35 (0.11 to 0.60)
Lateral step-down 0.13 (–0.02 to 0.29) 0.50 (0.14 to 0.95) 0.35 (0.07 to 0.64) 0.56 (0.36 to 0.77) 0.68 (0.39 to 0.97) 0.57 (0.30 to 0.84)
* Percent agreement and weighted kappa values are given for each rater.
Downloaded from www.jospt.org at on November 7, 2015. For personal use only. No other uses without permission.

higher percentage agreement compared
to the specific method, but weighted
kappa coefficients tended to be higher
for the specific method than the overall
method. The highest kappa coefficients
Copyright © 2007 Journal of Orthopaedic & Sports Physical Therapy®. All rights reserved.
evaluate clinician agreement for 2 func-
tional tasks commonly used to drive clin-
ical decision making for lower extremity
injuries.10,14 It was not our intention to
create a scoring system for all lower ex-
quality. Finally, raters were not given
detailed instruction for rating severity
of the movement deviation, such as us-
ing anatomical or joint angle reference
points. Intuitively, providing greater

were obtained for the lateral step-down
task using the specific method.
This study was conducted as a first
step toward developing a standardized
method of evaluating movement qual-
ity during lower extremity functional
tasks. In designing our study, we made
3 important decisions. First, we chose to
Journal of Orthopaedic & Sports Physical Therapy®
tremity functional tasks, which may re-
quire consideration of other movement
deviations. Second, we specifically chose
not to include measures associated with
balance (eg, touching the contralateral
foot to the floor) in our rating criteria.
In this way, our study purely investigat-
ed agreement in discerning movement
structure to rating severity of deviation
should assist in higher rater agreement.
We decided, however, that the first step
should be to understand the strengths
and limitations of current methods used
for visually analyzing movement.
As a general point, level of agreement
between and within raters was not high
for either method of evaluating move-

ment. However, caution must be taken

Rater 1
when comparing the rating methods and
Row
Poor Fair Good
Total interpreting the results of this study. The
number of rating categories was greater in
1  (0.1) 0  (0.7) 0  (0.2)
Poor 1 the specific method than the overall meth-
0 1 4
od to allow for a more responsive rating
1  (1.1) 11  (10.1) 2  (2.8) method. The difference in the number of
Fair 14
Rater 2

1 0 1
rating categories affects the kappa statis-
0  (0.8) 7  (7.2) 3  (2.0) tic, meaning that the rating methods are
Good 10
4 1 0 not directly comparable. Second, ratings
Column tended to be clustered (tight distribution)
2 18 5 25
Total for both rating methods. Such clustering
increases chance agreement and can lower
 observed frequencies (fo) = 0×1 + 1×0 + 4×0 + 1×1 + 0×11 + 1×2 + 4×0 + 1×7 + 0×3 = 10 the values for kappa statistics.16 Including
 expected frequencies (fe) = 0×0.1 + 1×0.7 + 4×0.2 +1×1.1 + 0×10.1 + 1×2 + 4×0.8 + 1× 7.2 + 0×3 = 15.8
subjects with greater movement compen-
weighted = 1 – (fo ⁄ fe) = 1 – (10 ⁄ 15.8) = 0.37 sations (eg, patients with lower extremity
pathology) might have resulted in higher
FIGURE 3. Example calculation of the weighted kappa statistic. Data presented are ratings from the unilateral squat kappa values for both rating methods.
task. On the top row of each cell are the observed (bold) and expected (in parentheses) frequencies. Expected fre-
Thirdly, kappa values are affected by the
quencies are calculated by multiplying the column total by the row total and dividing by the grand total. For example,
the expected frequency of observations in the “poor-poor” cell is calculated as follows: (2 × 1)/25. The number on the distribution of disagreements.16 If dis-
bottom row of each cell is the “weighting” assigned to an observation in that cell. The weighting matrix is configured agreements are distributed such that 1 rat-
such that greater penalty is given for larger discrepancies in agreement. For example, the value 4 for a “poor-good” er tends to rate deviations as more severe
combination indicates a higher penalty than the value 1 for “fair-good” combination. Note that perfect agreement than another rater, kappa values will be
receives a weighting of 0.
higher than if the disagreements are more

journal of orthopaedic & sports physical therapy | volume 37 | number 3 | march 2007 | 127
 [ research report
random. We observed that in the lateral
step-down task rater 1 typically scored
movement deviations less severely than
the other raters, whereas rater 2 scored
them as more severe, possibly inflating
the kappa values for this task. The use of
explicit guidelines for rating movement
deviation severity should improve consis-
tency across raters, eliminating the effect
of rater bias on the data. It is important
to remember that the primary concern
in using visual observation to evaluate
movement deviations is the avoidance of
Downloaded from www.jospt.org at on November 7, 2015. For personal use only. No other uses without permission.
large discrepancies in ratings that could
result in different treatment plans. A re-
view of our data demonstrates that there
was only 1 trial for the lateral step-down
task and 2 trials for the unilateral squat
task where rater pairs had polar-opposite
Copyright © 2007 Journal of Orthopaedic & Sports Physical Therapy®. All rights reserved.
ratings using the overall method (eg, good
versus poor). For the specific method, if
disagreement by 1 cell (eg, 10 versus 15)
was included in the calculation, percent
agreement ranged from 50% to 95% for
all rater pairs. Despite fairly low rater
agreement in this study, it is questionable
whether the disagreements would have
resulted in remarkably different treat-
Journal of Orthopaedic & Sports Physical Therapy®
ment approaches.
A few other studies have investigated
rater agreement in evaluating movement
during lower extremity functional tasks.
Because of differences in study design,
results from these studies cannot be di-
rectly compared to the current study, but
they indicate the status of visual analysis
development. Reimann et al15 described
interrater agreement for a single-leg hop
stabilization test in which subjects made
10 hops and were rated on landing and
balance errors. Almost all criteria were
related to performance or loss of balance
with the exception of 1 movement quality
rating in the balance error category (non-
dominant limb moving into greater than
30° flexion, extension, or abduction). A
lower intraclass correlation coefficient for
the balance score (0.74) than the landing
score (0.92) may have resulted from the
difficulty in evaluating movement qual-
ity. Harrison and colleagues8 investigated
clinician agreement for evaluating sin-
128 | march 2007 | volume 37 | number 3 | journal of orthopaedic & sports physical therapy
gle-leg standing ability in 78 uninjured
subjects and 17 patients. Trials were
scored as good, fair, and poor, according
to whether compensatory movements
were made at the trunk and extremities.
Guidelines were provided for judging
compensatory movements; for example,
poor grade meant there were 3 or more
compensatory movements and postural
sway fell outside the base of support. This
rating method, which is similar in struc-
ture to our overall method, resulted in a
weighted kappa coefficient of 0.70. The
larger subject pool, inclusion of patient
subjects (potentially increasing the dis-
tribution of ratings), and use of specific
guidelines for rating severity of move-
ment deviation may all have contributed
to the high kappa value. Bjorklund and
colleagues1 reported interrater agreement
for observational analysis of subjects per-
forming a wide variety of lower extrem-
ity functional tasks, including running
and hopping. Each task had 5 different
criteria and each criterion had 2 choices.
Using an unweighted kappa statistic, in-
terrater coefficients ranged between 0.62
and 0.78 for all tests, whereas intrarater
coefficients ranged from 0.34 and 0.64.
Not unexpectedly, intrarater agreement
was worse for tests that involved subjec-
tive judgments of movement quality, in-
cluding the “springiness” and “rhythm”
during hops and jumps. Finally, Piva et
al14 reported an interrater kappa coeffi-
cient of 0.67 (95% confidence intervals
[CI]: 0.58-0.76) for movement quality
assessment during a lateral step-down
task performed by individuals with
patellofemoral pain. The 5 rating crite-
ria included trunk, pelvis, and knee po-
sition, and use of arms for balance, and
loss of balance. Criteria were scored di-
chotomously (1 point if the deviation was
present), except knee position, for which
deviation severity was rated based on
anatomical reference points. Total scores
were then grouped into 1 of 3 categories.
Results of this study indicate that col-
lapsing the ratings of individual body
segments into categories, providing fewer
rating choices for deviation severity, and
]
using anatomical reference points assist
rater agreement.
The main limitations of this study that
merit discussion are the use of symbols
for scoring (specific method), inclusion of
only uninjured subjects, and the distribu-
tion of responses for both rating meth-
ods. We arbitrarily chose 3 easily written
symbols to denote severity of movement
deviation. It is possible that raters may
have confused the different symbols,
which would have affected our results;
but because each of the raters assisted in
developing the scoring methods and par-
ticipated in pilot testing, it is unlikely that
confusion between the symbols skewed
our results. In retrospect, a more intui-
tive scoring method should have been
implemented, especially if the ultimate
goal was to develop a standardized scor-
ing method for widespread use in clinical
practice. We also tested uninjured sub-
jects, which resulted in tight clustering of
the ratings. It is possible that including a
patient population would have provided
a better indication of agreement across a
wide range of movement patterns; how-
ever, important points must be made to
this regard. Observational analysis of
movement quality is performed in re-
habilitation and as part of screenings to
identify individuals that are predisposed
to lower extremity injury. Our study re-
sults have direct relevance for the latter
clinical application. Furthermore, Piva et
al14 reported that for patients with patel-
lofemoral pain who performed a lateral
step-down task only 17% of the move-
ment ratings fell into the poor category.
This is comparable to the 11.5% poor
ratings for the lateral step-down task in
our study. It is possible that movement
patterns in either injured or uninjured
populations tend to be acceptable or have
minor deviations. Assuming that very se-
vere movement deviations would be eas-
ily discriminated, the challenge may be to
develop standardized scoring methods to
discern minor movement deviations.
Continued research is needed to de-
velop and validate standardized methods
for observational analysis of movement
 quality. Future efforts should include
stricter criteria with explicit instruction
(eg, knee moves to the inside of the great
toe) as a means to improve agreement be-
tween raters. In addition, rating catego-
ries should be limited to as few choices
as possible—even dichotomous—if it is
not necessary to have a more sensitive
measure (ie, more categories) to quantify
movement deviation severity. Intuitive
documentation methods, such as using
“+/++/+++” to denote severity of move-
ment deviation or a 0/1 scoring system
Downloaded from www.jospt.org at on November 7, 2015. For personal use only. No other uses without permission.
for dichotomous rating would also be
beneficial. Analysis methods that consist
of ratings for individual body segments,
strict rating criteria, the fewest number
of rating categories, and the inclusion
of more easily observed balance criteria
Copyright © 2007 Journal of Orthopaedic & Sports Physical Therapy®. All rights reserved.
(eg, stepping down with contralateral
leg) should raise rater agreement to ac-
ceptable levels. Once reliable evaluation
methods are developed, validation of rat-
ings by comparing to objective measures
(eg, motion capture data) or clinical out-
comes will be necessary.
CONCLUSION
Journal of Orthopaedic & Sports Physical Therapy®
R
ating movement at individual
body segments appears to result
in agreement among raters that is
better than chance. Neither the overall
method nor the specific method of move-
ment analysis produced high rater agree-
ment, indicating a need to develop more
explicit criteria for rating movement de-
viation severity.
journal of orthopaedic & sports physical therapy | volume 37 | number 3 | march 2007 |
ACKNOWLEDGEMENTS
T
he authors acknowledge physi-
cal therapists Chu Soh and Mary
Gregory for assisting with the devel-
opment of this project. In addition, the
authors acknowledge Heidi Betz, Yadira
DeJesus, and Michelle Sanchez for their
assistance with data processing. t
references
1. B jorklund K, Skold C, Andersson L, Dalen N. Re-
liability of a criterion-based test of athletes with
knee injuries; where the physiotherapist and
the patient independently and simultaneously
assess the patient’s performance. Knee Surg
Sports Traumatol Arthrosc. 2006;14:165-175.
2. Cohen J. Weighted kappa: nominal scale agree-
ment with provision for scaled disagreement or
partial credit. Psychol Bull. 1968;70:213-220.
3. Cook G, Burton L, Hoogenboom B. Pre-par-
ticipation screening: the use of fundamental
movements as an assessment of function-Part I.
North Am J Sports Phys Ther. 2006;1:62-72.
4. Crewson PE. Reader agreement studies. AJR Am
J Roentgenol. 2005;184:1391-1397.
5. Ford KR, Myer GD, Smith RL, Vianello RM, Seiwert
SL, Hewett TE. A comparison of dynamic coronal
plane excursion between matched male and
female athletes when performing single leg land-
ings. Clin Biomech (Bristol, Avon). 2006;21:33-40.
6. Griffin LY, Agel J, Albohm MJ, et al. Noncontact
anterior cruciate ligament injuries: risk factors
and prevention strategies. J Am Acad Orthop
Surg. 2000;8:141-150.
7. Hardcastle P, Nade S. The significance of
the Trendelenburg test. J Bone Joint Surg Br.
1985;67:741-746.
8. Harrison EL, Duenkel N, Dunlop R, Russell G.
@
Evaluation of single-leg standing following ante-
rior cruciate ligament surgery and rehabilitation.
Phys Ther. 1994;74:245-252.
9. H  ewett TE, Myer GD, Ford KR, et al. Biome-
chanical measures of neuromuscular control
and valgus loading of the knee predict anterior
cruciate ligament injury risk in female ath-
letes: a prospective study. Am J Sports Med.
2005;33:492-501.
10. Kibler WB, Press J, Sciascia A. The role of
core stability in athletic function. Sports Med.
2006;36:189-198.
11. Mascal CL, Landel R, Powers C. Management
of patellofemoral pain targeting hip, pelvis, and
trunk muscle function: 2 case reports. J Orthop
Sports Phys Ther. 2003;33:647-660.
12. McLean SG, Huang X, van den Bogert AJ. As-
sociation between lower extremity posture at
contact and peak knee valgus moment during
sidestepping: implications for ACL injury. Clin
Biomech (Bristol, Avon). 2005;20:863-870.
13. McLean SG, Lipfert SW, van den Bogert AJ.
Effect of gender and defensive opponent on
the biomechanics of sidestep cutting. Med Sci
Sports Exerc. 2004;36:1008-1016.
14. Piva SR, Fitzgerald K, Irrgang JJ, et al. Reliability
of measures of impairments associated with
patellofemoral pain syndrome. BMC Musculosk-
elet Disord. 2006;7:33.
15. Riemann BL, Caggiano NA, Lephart S. Examina-
tion of a clinical method of assessing postural
control during a functional performance task. J
Sport Rehabil. 1999;8:171-183.
16. Sim J, Wright CC. The kappa statistic in reliabil-
ity studies: use, interpretation, and sample size
requirements. Phys Ther. 2005;85:257-268.
17. Willson JD, Dougherty CP, Ireland ML, Davis
IM. Core stability and its relationship to lower
extremity function and injury. J Am Acad Orthop
Surg. 2005;13:316-325.
18. Zeller BL, McCrory JL, Kibler WB, Uhl TL.
Differences in kinematics and electromyo-
graphic activity between men and women dur-
ing the single-legged squat. Am J Sports Med.
2003;31:449-456.
more information
www.jospt.org
129