Orthopaedic Advances

Measuring Surgical Skills in

Simulation-based Training

Kivanc Atesok, MD, MSc
Richard M. Satava, MD
J. Lawrence Marsh, MD
Shepard R. Hurwitz, MD
Downloaded from http://journals.lww.com/jaaos by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdgGj2MwlZLeI= on 07/02/2021
Abstract
Simulation-based surgical skills training addresses several concerns
associated with the traditional apprenticeship model, including patient
safety, efficient acquisition of complex skills, and cost. The surgical
specialties already recognize the advantages of surgical training
using simulation, and simulation-based methods are appearing in
surgical education and assessment for board certification. The

necessity of simulation-based methods in surgical education along

with valid, objective, standardized techniques for measuring learned
skills using simulators has become apparent. The most commonly
used surgical skill measurement techniques in simulation-based
training include questionnaires and post-training surveys, objective
structured assessment of technical skills and global rating scale of
performance scoring systems, structured assessments using video
recording, and motion tracking software. The literature shows that the
application of many of these techniques varies based on investigator
preference and the convenience of the technique. As simulators
become more accepted as a teaching tool, techniques to measure skill
proficiencies will need to be standardized nationally and
internationally.

I ncorporating simulation-based sur-

From the Department of Orthopaedic
Surgery, University of Alabama at
Birmingham, Birmingham, AL
(Dr. Atesok), the Department of
Surgery, University of Washington
Medical Center, Seattle, WA
(Dr. Satava), the Department of
Orthopaedics and Rehabilitation,
University of Iowa, Iowa City, IA
(Dr. Marsh), and the Department of
Orthopaedics, University of North
Carolina at Chapel Hill, Chapel Hill,
NC (Dr. Hurwitz).
J Am Acad Orthop Surg 2017;25:
665-672
DOI: 10.5435/JAAOS-D-16-00253
Copyright 2017 by the American
Academy of Orthopaedic Surgeons.
gical skills training in the educa-
tional curriculum addresses the current
challenges trainees face, including
acquiring complex skills efficiently
despite work-hour restrictions, cost
pressures, and policies intended to
reduce patient waiting times.1,2
Simulation-based training also enables
the implementation of the principles of
so-called proficiency-based training,
which focuses on assisting trainees to
reach a specified level of performance
and achieve a uniform set of skills
required to perform certain proce-
dures. Quantitative assessment of the
level of proficiency that uses standard
and objective metric measurements is
critically important in improving the
quality of surgical education and in
training professional, competent sur-
geons.3 As in all metric systems, the
measurement tools used in the assess-
ment of surgical proficiency need to be
practical, usable, objective, valid, and
reliable to be accepted as standard.
Tools used in assessing surgical skills
are listed in Table 1.
Measurement Tools Used in
Simulation-Based Surgical
Skills Training
Questionnaires
Questionnaires are designed to gen-
erate feedback from trainees regard-
ing their personal feeling of comfort
or knowledge level in performing a
surgical procedure. Although ques-
tionnaires can be practical, low-cost
assessment tools, they have several

October 2017, Vol 25, No 10 665

Copyright ª the American Academy of Orthopaedic Surgeons. Unauthorized reproduction of this article is prohibited.
Measuring Surgical Skills in Simulation-based Training

Table 1
Summary of Measurement Tools Currently Used for the Assessment of Surgical Skills
Measurement Tool Objectivity Validity Reliability Comments

Questionnaires YYY None YY Susceptible to poor instrument, subject, and

situational reliability because of the subjective
nature of the tool
OSATS [[ [[[ [[ Most commonly used objective and valid tool with
high reliability
GRS [ [ [ Valid adjunct to OSATS with qualitative assessment
parameters
Motion tracking [[[ [[ Unknown Highly objective tool; although shown to be valid,
and analysis quality of overall task performance might not be
assessed sufficiently; insufficient data on reliability
Video recording [[ [[ [[ Aids in assessment of skills using tools such as
OSATS and GRS at the time most convenient for the
rater
C-SATS [[ [[ [[ An adjunct to video-recorded assessment to further
reduce assessment time and cost
Direct objective [[[ [[ [[ Highly objective; parameters such as task completion
metric tools time must be combined with quality of task
completion such as knot-tying time and knot
strength

C-SATS = Crowd-Sourced Assessment of Technical Skills, GRS = Global Rating Scale, OSATS = Objective Structured Assessment of Technical
Skills, YY = low,YYY = very low,[ = moderate to high,[[ = high,[[[ = very high

inherent shortcomings, including sub-
jectivity and unfeasibility in terms of
standardization. Furthermore, vali-
dating questionnaires can be chal-
lenging, because they evaluate
subjective measures that can be biased
by many variables related to the sub-
jects’ self-assessments of qualitative
parameters. Most published studies
in which comfort or knowledge ques-
tionnaires were used as proficiency
measures of surgical procedures re-
port that such questionnaires are
not validated instruments.4,5 Thus, a
questionnaire is not a suitable mea-
surement tool for validated, standard,
and metric assessments of surgical
competence.
Objective Structured
Assessment of Technical
Skills
The Objective Structured Assessment
of Technical Skills (OSATS) was the
first assessment tool that made possible
the quantitative measurement of surgi-
cal skill or task performance in surgical
simulation.6 The OSATS is done by
independent observers who evaluate
the trainee’s performance using a
checklist consisting of a set of specific
surgical maneuvers that have been
deemed essential elements of the pro-
cedure, such as appropriate placement
of plate on bone using a C-arm and
securing proximal and distal fixation1
(Figure 1). Accordingly, it is critical to
train the observers who perform the
scoring so that interrater (observer)
reliability is .0.80 (ie, has almost
perfect agreement between observers)
to achieve unbiased results.
The OSATS methodology was de-
signed for performance evaluation after
the completion of a training session;
however, it can be used during training
to standardize formative feedback.
When the trainer (faculty member) uses
a checklist during the novice practice
sessions, the skills or tasks can be
monitored in real time. When a check-
list item is not performed by the trainee,
standardized, formative feedback can
be given. Training and assessment are
two sides of the same coin; thus,

Dr. Satava or an immediate family member serves as a paid consultant to Medtronic Minimally Invasive Therapies; has received research or
institutional support from Intuitive Surgical; and serves as a board member, owner, officer, or committee member of the American College of
Surgeons. Dr. Marsh or an immediate family member has received royalties from Zimmer Biomet and Wright Medical Group N.V.; has stock
or stock options held in FxRedux; and serves as a board member, owner, officer, or committee member of the American Board of
Orthopaedic Surgery and the National Board of Medical Examiners. Dr. Hurwitz or an immediate family member serves as a board member,
owner, officer, or committee member of the American Board of Orthopaedic Surgery, the Orthopaedic Research and Education Foundation,
and the Orthopaedic Trauma Association. Neither Dr. Atesok nor any immediate family member has received anything of value from or has
stock or stock options held in a commercial company or institution related directly or indirectly to the subject of this article.

666 Journal of the American Academy of Orthopaedic Surgeons

Copyright ª the American Academy of Orthopaedic Surgeons. Unauthorized reproduction of this article is prohibited.
 Kivanc Atesok, MD, MSc, et al

Figure 1

The Objective Structured Assessment of Technical Skills checklist for carpal tunnel release. (Adapted with permission from
Van Heest A, Kuzel B, Agel J, et al: Objective structured assessment of technical skill in upper extremity surgery. J Hand
Surg 2012;37[2]:332-337.)

training can be greatly enhanced by
formative feedback, especially in train-
ing to proficiency.
It is important to note that an OSATS
checklist reports whether each and
every essential step of a surgical pro-
cedure was completed; the tool does
not measure quality or surgical finesse.7
Global Rating Scale
The Global Rating Scale (GRS) is
another commonly used surgical
skills assessment tool used to measure
characteristic surgical behaviors (ie,
surgical finesse) during the perfor-
mance of any given procedure (eg,
respect for soft tissues, fluidity of
movements, familiarity with the

October 2017, Vol 25, No 10 667

Copyright ª the American Academy of Orthopaedic Surgeons. Unauthorized reproduction of this article is prohibited.
Measuring Surgical Skills in Simulation-based Training
instruments)1,8 (Figure 2). Although
the GRS was developed to comple-
ment OSATS, some researchers
include this assessment tool as a
component of OSATS. Because the
surgical skill parameters measured
using the GRS have characteristic
differences from items included in
the OSATS checklist, it would be
wise to think of the GRS and OSATS
as separate measurement tools.
Typically, the GRS uses a rating scale
such as the Likert scale and measures
surgical behaviors in general.
Therefore, the GRS arguably pro-
vides a comprehensive assessment,
which includes objective and sub-
jective criteria and measures non-
technical cognitive skills (eg, decision
making, judgment). Adding sub-
jective criteria to any measurement
tool that uses rating scales but does
not use well-defined yes-or-no
checklists, however, results in the
limitations associated with sub-
jectivity, including ambiguity, poor
interrater reliability, and bias.
In a recent study, Bernard et al7
reported that OSATS checklist scores
showed strong interrater reliability
(.0.8) between the evaluators in
assessing technical skills pertaining
to shoulder surgical approaches. The
GRS scores were found to be mod-
erately reliable (0.75) between eval-
uators, however. The results of this
study also showed that the OSATS
checklist and GRS scores correlated
with the trainees’ levels of experi-
ence. This finding supports the val-
idity of these measurement tools in
differentiating between the skills of
novice and experienced trainees in
performing surgical approaches to
the shoulder.
Motion Tracking and
Analysis
Objective assessment of performance
with simulators requires metrics to
provide accurate measurement of
surgical skills. The most commonly
668
Copyright ª the American Academy of Orthopaedic Surgeons. Unauthorized reproduction of this article is prohibited.
used metric measurement methods
include task completion time and
accuracy. Task completion time and
error can provide sufficient data to
differentiate between a novice and an
expert in the simulated performance
of surgical skills. Detection of differ-
ences between the intermediate level
of expertise and the novice or expert
level can be challenging, however,
because the latter two parameters do
not supply metric information about
the fluidity of hand movements when
performing a task. Motion tracking
and analysis appears to be an objec-
tive and valid tool for assessing sur-
gical skills in terms of precision and
economy of movement during the
performance of surgical proce-
dures.9,10 Motion tracking systems
can be mounted to surgical tools and
attached to or worn on the hands as
sensors. The movements of these
sensors are recorded as three-
dimensional coordinates to measure
a variety of motions, including the
total path length traced by each
sensor and the number of trans-
lational or rotational movements.11
The main disadvantage of these
systems is that they require attaching
extra devices to surgical tools or
wearing equipment on the arms or
hands, which can be cumbersome
and impractical for the assessment of
surgical skills. This method does not
always require such cumbersome
implementations, especially in
arthroscopic procedures. Motion
analysis systems can be built into a
simulator to track and analyze
instrument tip trajectory data.12
Howells et al9 showed the validity of
a motion analysis system in its ability
to differentiate between subjects
with different expertise levels in
arthroscopic skills. The authors re-
corded the time taken, total path
length, and number of movements
used when performing simulated
arthroscopic tasks using a shoulder
simulator equipped with an electro-
magnetic motion tracking system.
Journal of the American Academy of Orthopaedic Surgeons
They found significant differences
between the performances of the
surgeon and nonsurgeon groups
(P , 0.0001) and between senior and
junior surgeons (P , 0.05).
Despite the increasing availability
of simulators that can track and
analyze motion and the positive
effects of this method on the objec-
tive assessment of proficiency in
surgical skills, the effect of these
metrics on trainees’ skill transfer to
the operating room has yet to be
proved.13 It seems unlikely that
motion analysis will be used widely
in the actual operating room set-
ting. Motion tracking and analysis
will likely remain a research tool for
selected laboratory-based simula-
tion studies.
Video Recording
Video recording of the operation for
later assessment of surgical skills has
several advantages over the currently
used assessment methods, including
real-time OSATS testing, during
which an observer must be ready in
the operating room to rate a trainee’s
performance on a checklist. Video-
based feedback is a practical method
that enables the assessment of sur-
gical performance using the same
measurement tools as the OSATS or
the GRS at the time that is most
convenient for the rater14 (Figure 3).
Multiple raters can examine the
same video recording and score the
performance, which may be effective
in reducing bias. Video recordings
are edited after the procedure, and
unnecessary parts of the recording
are cut; thus, the evaluation of sur-
gical performance using video
recordings also may reduce the time
needed to assess the complete pro-
cedure. Although the process of ed-
iting may take extra time,15 the
potential time saved by enabling
multiple raters to assess the edited
recordings arguably far outweighs
the time spent on editing.
 Kivanc Atesok, MD, MSc, et al

Figure 2

The Global Rating Scale score sheet for carpal tunnel release. (Adapted with permission from Van Heest A, Kuzel B, Agel J,
et al: Objective structured assessment of technical skill in upper extremity surgery. J Hand Surg 2012;37[2]:332-337.)

October 2017, Vol 25, No 10 669

Copyright ª the American Academy of Orthopaedic Surgeons. Unauthorized reproduction of this article is prohibited.
Measuring Surgical Skills in Simulation-based Training

Figure 3 cine robot-assisted urinary bladder

closures. The authors compared the
assessments of 7 expert raters with
those of 487 crowd workers re-
cruited from Amazon Mechanical
Turk. The expert surgeon graders
took 14 days to complete the scoring
of video-recorded performances
using the GRS, whereas the crowd
workers completed the assessments
of 12 videos in 4 hours and
28 minutes. Each rater from Amazon
Mechanical Turk was paid $0.75 per
video. Concordance between the
Images demonstrating a simulated fracture fixation session. A, Photograph
showing a trainee equipped with a head-mounted camera for the video recording surgeon graders and the crowd
of performance. B, Video screen capture showing the view from the head- workers was 0.93 (the Cronbach
mounted camera during the placement of a Kirschner wire. (Reproduced with Alpha). Interrater (observer) reli-
permission from Karam M, Thomas G, Koehler D, et al: Surgical coaching from ability among the surgeon graders
head-mounted video in the training of fluoroscopically guided articular fracture
surgery. J Bone Joint Surg Am 2015;97[12]:1031-1039.) was 0.89.
The surgical skills assessment tools
currently in use require too much
Video recordings from cameras can serve many purposes, such as life- time and too many resources and
positioned in the operating room or long learning with self-assessment for therefore are not practical for fre-
simulation centers can be valuable improvement and quality assurance quent use or scaling to larger stud-
additions to the surgical skills for risk management, or privileging ies.18,19 Using crowdsourcing to
assessment of almost any type of and credentialing. As technology assess surgical skills may reduce the
procedure, but arthroscopic opera- improves, video capture likely will burden of the assessments and rater
tions in particular are well suited for play a greater role in the formative and bias while increasing the account-
this type of assessment, because the summative assessment of surgical skill. ability of outcomes by soliciting
monitor output can be saved auto- contributions from a large online
matically as a video file.16 Jabbour community rather than from pro-
and Sidman16 demonstrated the Crowd-Sourced Assessment fessionals who are related to a par-
feasibility of using time-synced mul- of Technical Skills ticular project in some way.18,19
ticamera videos to show the instru- The term crowdsourcing has been
ment handling and the surgical field defined as “the practice of obtaining
from the arthroscopic camera video information or input into a task or Direct Objective Metric
recordings. The authors stated that project by enlisting the services of a Measures
the “mean duration of OSATS vid- large number of people, either paid Directly and objectively measuring a
eos was 11 minutes and 20 seconds, or unpaid, typically via the Inter- concrete aspect of a skill using uni-
which was significantly less than the net.”17 Crowd-Sourced Assessment versal metric measurements holds
time needed for an expert to be of Technical Skills (C-SATS) is an promise for improving reliability,
present at the administration of each emerging adjunct for the video- validity, clinical relevance, and
30-minute OSATS (P , 0.001).” recorded evaluation of surgical applicability in large-scale studies or
Video recording is valuable not skills. In this method, video-recorded high-stakes board examinations
only for the initial training of a novice surgical performance can be assessed while reducing time and expense.
or for training an experienced sur- (using the OSATS or the GRS) by Examples of such parameters include
geon in a new procedure, but also for online crowds of raters who are de- the mechanical strength of a knot or a
the maintenance of certification centralized, anonymous, and inde- fracture fixation construct, the dis-
in periodic assessments. The public pendent; some observers may not tance travelled when navigating a
increasingly demands more oversight have received medical training.18 wire to a certain location such as the
to ensure the quality of surgical per- Holst et al19 studied the validity of center of the femoral head, the accu-
formance. A standardized quantitative C-SATS in assessing the surgical racy of reduction, or the time to
review of video-recorded procedures performance of 12 surgeons in por- completion of a skill task. Research

670 Journal of the American Academy of Orthopaedic Surgeons

Copyright ª the American Academy of Orthopaedic Surgeons. Unauthorized reproduction of this article is prohibited.

has demonstrated the value of using
these parameters in surgical skills
assessments when compared with
accustomed methods such as
OSATS.20,21
In a simulated intra-articular frac-
ture reduction model, Anderson
et al20 showed that OSATS did not
correlate well with the actual frac-
ture reduction measured using three-
dimensional digital models of the
final reductions of articular surfaces.
Similarly, in a distal radius fracture
fixation model, Putnam et al21
showed that the biomechanical
strength of the fracture construct did
not correlate with medical knowl-
edge tests and OSATS scores. These
data strongly suggest that the pre-
viously described direct and objec-
tive metric measurement parameters
are critically important adjuncts to
more commonly used techniques,
such as the OSATS and the GRS, in
the assessment of surgical skills.
Future Work
Currently, the consensus in ortho-
paedic leadership—and in surgical
subspecialties in general—is that a
paradigm shift that integrates
simulation-based surgical skills
training into curricula is necessary
in surgical education. Furthermore,
simulation-based assessment in high-
stakes surgery board examinations
across all surgical subspecialties is
also foreseeable. The measurement
techniques that will determine the
level of proficiency are not yet well
defined, however. In a systematic
review, Slade Shantz et al22 studied
the internal validity of arthroscopic
simulators and aimed to document
whether any standard validation
protocols exist. The authors re-
ported excessive heterogeneity in
the literature concerning perfor-
mance metrics used in assessments
and noted that the simulators can
discriminate between novices and
October 2017, Vol 25, No 10
Copyright ª the American Academy of Orthopaedic Surgeons. Unauthorized reproduction of this article is prohibited.
experts. Nevertheless, questions
remain regarding the ability of
simulators to discriminate between
novice and intermediate proficiency
levels. The main reason for the
inability to discriminate between
relatively close proficiency levels
could be the use of nonstandardized
measurement techniques that are
not sensitive enough to quantify
such differences.
As the use of simulators becomes
more commonplace, it will be critical
to define a full–life-cycle, simulation-
based surgical training curriculum
using proficiency-based progression
methodology (ie, training to a
benchmark that has been established
by expert performance)23 along with
objective, reliable, and valid mea-
surement protocols that are stan-
dardized across all training programs
nationwide. Perhaps such measure-
ment protocols will need to combine
more objective techniques such as
OSATS, motion analysis, and direct
metrics with video recording and
C-SATS. Although the curriculum
may be subject to change based on
the country in which it is used, it will
be necessary to achieve consistency in
measurement protocols internation-
ally to set a standard and ensure
efficiency in communicating the level
of surgical proficiency.
Summary
Simulation-based skills training and
assessment are increasingly incorpo-
rated into surgical education and
certification processes. Measurement
techniques to quantify the level of
proficiency in the performance of
surgical procedures will be the key
element in the success of this para-
digm shift in surgical education.
Some of the current measurement
methods used to assess surgical skills
include the OSATS, the GRS, motion
tracking, video recording, C-SATS,
and direct or objective metric mea-
Kivanc Atesok, MD, MSc, et al
sures.24 These methods are used
alone or in combination based on the
preferences of each research group
or institution. Therefore, heteroge-
neity exists in the literature con-
cerning the available evidence
needed to draw conclusions. There is
a need to define a standard, full–life-
cycle, simulation-based surgical
education curriculum along with
measurement protocols using reli-
able, valid, and objective metrics and
to adopt a proficiency-based pro-
gression methodology.24
References
Evidence-based Medicine: Levels of
evidence are described in the table of
contents. In this article, references 13
and 23 are level I studies. References
4-9, 11, 14, 15, and 19-21 are level II
studies. References 10, 12, and 22
are level III studies. Reference 16 is a
level IV study. References 1-3, 18,
and 24 are level V expert opinion.
References printed in bold type are
those published within the past 5
years.
1. Atesok K, Mabrey JD, Jazrawi LM, Egol
KA: Surgical simulation in orthopaedic
skills training. J Am Acad Orthop Surg
2012;20(7):410-422.
2. Atesok K, Satava RM, Van Heest A, et al:
Retention of skills after simulation-based
training in orthopaedic surgery. J Am Acad
Orthop Surg 2016;24(8):505-514.
3. Gallagher AG: Metric-based simulation
training to proficiency in medical
education: What it is and how to do it.
Ulster Med J 2012;81(3):107-113.
4. Beth Grossman L, Komatsu DE,
Badalamente MA, Braunstein AM, Hurst
LC: Microsurgical simulation exercise for
surgical training. J Surg Educ 2016;73(1):
116-120.
5. Monod C, Voekt CA, Gisin M, Gisin S,
Hoesli IM: Optimization of competency in
obstetrical emergencies: A role for
simulation training. Arch Gynecol Obstet
2014;289(4):733-738.
6. Reznick R, Regehr G, MacRae H,
Martin J, McCulloch W: Testing technical
skill via an innovative “bench station”
examination. Am J Surg 1997;173(3):
226-230.
671
Measuring Surgical Skills in Simulation-based Training
7. Bernard JA, Dattilo JR, Srikumaran U,
Zikria BA, Jain A, LaPorte DM: Reliability
and validity of 3 methods of assessing
orthopedic resident skill in shoulder
surgery. J Surg Educ 2016;73(6):
1020-1025.
8. Alvand A, Logishetty K, Middleton R,
et al: Validating a global rating scale to
monitor individual resident learning
curves during arthroscopic knee
meniscal repair. Arthroscopy 2013;29
(5):906-912.
9. Howells NR, Brinsden MD, Gill RS, Carr
AJ, Rees JL: Motion analysis: A validated
method for showing skill levels in
arthroscopy. Arthroscopy 2008;24(3):
335-342.
10. Mason JD, Ansell J, Warren N, Torkington J:
Is motion analysis a valid tool for assessing
laparoscopic skill? Surg Endosc 2013;27(5):
1468-1477.
11. Clinkard D, Holden M, Ungi T, et al: The
development and validation of hand
motion analysis to evaluate competency in
central line catheterization. Acad Emerg
Med 2015;22(2):212-218.
12. Tashiro Y, Miura H, Nakanishi Y, Okazaki K,
Iwamoto Y: Evaluation of skills in
arthroscopic training based on trajectory
and force data. Clin Orthop Relat Res
2009;467(2):546-552.
672
Copyright ª the American Academy of Orthopaedic Surgeons. Unauthorized reproduction of this article is prohibited.
13. Stefanidis D, Yonce TC, Korndorffer JR Jr,
Phillips R, Coker A: Does the incorporation
of motion metrics into the existing FLS
metrics lead to improved skill acquisition
on simulators? A single blinded,
randomized controlled trial. Ann Surg
2013;258(1):46-52.
14. Dath D, Regehr G, Birch D, et al: Toward
reliable operative assessment: The
reliability and feasibility of videotaped
assessment of laparoscopic technical skills.
Surg Endosc 2004;18(12):1800-1804.
15. Karam MD, Thomas GW, Koehler DM,
et al: Surgical coaching from head-mounted
video in the training of fluoroscopically
guided articular fracture surgery. J Bone
Joint Surg Am 2015;97(12):1031-1039.
16. Jabbour N, Sidman J: Assessing instrument
handling and operative consequences
simultaneously: A simple method for creating
synced multicamera videos for endosurgical
or microsurgical skills assessments. Simul
Healthc 2011;6(5):299-303.
17. Oxford Living Dictionaries. Online volume
2017.
18. Lendvay TS, White L, Kowalewski T:
Crowdsourcing to assess surgical skill.
JAMA Surg 2015;150(11):1086-1087.
19. Holst D, Kowalewski TM, White LW, et al:
Crowd-sourced assessment of technical
Journal of the American Academy of Orthopaedic Surgeons
skills: Differentiating animate surgical skill
through the wisdom of crowds. J Endourol
2015;29(10):1183-1188.
20. Anderson DD, Long S, Thomas GW,
Putnam MD, Bechtold JE, Karam MD:
Objective structured assessments of
technical skills (OSATS) does not assess the
quality of the surgical result effectively.
Clin Orthop Relat Res 2016;474(4):
874-881.
21. Putnam MD, Kinnucan E, Adams JE, Van
Heest AE, Nuckley DJ, Shanedling J: On
orthopedic surgical skill prediction: The
limited value of traditional testing. J Surg
Educ 2015;72(3):458-470.
22. Slade Shantz JA, Leiter JR, Gottschalk T,
MacDonald PB: The internal validity of
arthroscopic simulators and their
effectiveness in arthroscopic education.
Knee Surg Sports Traumatol Arthrosc
2014;22(1):33-40.
23. Angelo RL, Ryu RK, Pedowitz RA, et al: A
proficiency-based progression training
curriculum coupled with a model simulator
results in the acquisition of a superior
arthroscopic Bankart skill set. Arthroscopy
2015;31(10):1854-1871.
24. Atesok K, MacDonald P, Leiter J, et al:
Orthopaedic education in the era of surgical
simulation: Still at the crawling stage.
World J Orthop 2017;8(4):290-294.