You are on page 1of 10

Surg Endosc (2013) 27:854863 and Other Interventional Techniques

DOI 10.1007/s00464-012-2524-9

A study of psychomotor skills in minimally invasive surgery:

what differentiates expert and nonexpert performance
Erlend Fagertun Hofstad Cecilie Vapenstad
Magdalena Karolina Chmarra Thomas Lang

Esther Kuhry Ronald Marvik

Received: 17 February 2012 / Accepted: 27 July 2012 / Published online: 6 October 2012
Springer Science+Business Media, LLC 2012

Abstract intermediates (31500 MIS), and novices (no experience in

Background A high level of psychomotor skills is MIS). The participants performed the labyrinth task in the
required to perform minimally invasive surgery (MIS) D-box Basic simulator (D-Box Medical, Lier, Norway).
safely. To assure high quality of skills, it is important to be The task required bimanual maneuvering and threading a
able to measure and assess these skills. For that, it is needle through a labyrinth of 10 holes. Nine motion-related
necessary to determine aspects that indicate the difference metrics were used to assess the MIS skills of each
between performances at various levels of proficiency. participant.
Measurement and assessment of skills in MIS are best done Results Experts (n = 7) and intermediates (n = 14) per-
in an automatic and objective way. The goal of this study formed significantly better than the novices (n = 11) in
was to investigate a set of nine motion-related metrics for terms of time and parameters measuring the amount of
their relevance to assess psychomotor skills in MIS during instrument movement. The experts had significantly better
the performance of a labyrinth task. bimanual dexterity, which indicated that they made more
Methods Thirty-two surgeons and medical students were simultaneous movements of the two instruments compared
divided into three groups according to their level of to the intermediates and novices. The experts also per-
experience in MIS; experts ([500 MIS procedures), formed the task with a shorter instrument path length with
the nondominant hand than the intermediates.
Conclusions The surgeons performance in MIS can be
E. F. Hofstad (&)  C. Vapenstad  T. Lang distinguished from a novice by metrics such as time and
Department of Medical Technology, SINTEF Technology path length. An experienced surgeon in MIS can be dif-
and Society, PB 4760 Sluppen, 7465 Trondheim, Norway ferentiated from a less experienced one by the higher
ability to control the instrument in the nondominant hand
M. K. Chmarra and the higher degree of simultaneous (coordinated)
Department of Circulation and Medical Imaging, Faculty movements of the two instruments.
of Medicine, Norwegian University of Science and Technology
(NTNU), Trondheim, Norway
Keywords Assessment  Box trainer  Minimally
E. Kuhry  R. Marvik invasive surgery  Motion analysis  Psychomotor skills 
Department of Cancer Research and Molecular Medicine, Training
Faculty of Medicine, Norwegian University of Science
and Technology (NTNU), Trondheim, Norway

E. Kuhry  R. Marvik An increasing part of surgical interventions is done by

Department of Surgery, St. Olavs Hospital, Trondheim minimally invasive surgery (MIS), such as laparoscopy [1].
University Hospital, Trondheim, Norway Compared to traditional open surgery, a new set of psy-
chomotor skills is required to perform MIS safely, in
E. Kuhry  R. Marvik
National Center for Advanced Laparoscopic Surgery, St. Olavs addition to skills related to knowledge and judgment [2, 3].
Hospital, Trondheim University Hospital, Trondheim, Norway Surgical simulators, both virtual reality (VR) and analog

Surg Endosc (2013) 27:854863 855

box trainers, are already an important part of surgical validated to ensure relevant skills, in this case for mini-
training [4, 5]. Training on simulators outside of the mally invasive surgery, are trained and assessed. To our
operating room is risk-free for the patient. It has the knowledge, such a validation has not previously been
potential to be fair and reliable for all participants with published for a similar setup as the one presented here: a
identical setups, and to objectively assess surgical perfor- labyrinth task in an analog box trainer. Laparoscopic sur-
mance. However, it is essential to make sure that the skills geons with various levels of experience participated in the
learned on the simulator actually improve surgical perfor- study, along with medical students and interns. The
mance, by establishing predictive validity [6, 7]. To assessment was based on the time to complete the task and
determine predictive validity and, in general, to assess MIS eight metrics calculated from the movements of the lapa-
skills, a participants performance can be evaluated, for roscopic instruments. The instruments were tracked by an
example, during surgery using assessment methods such as electromagnetic tracking system.
the Global Operative Assessment of Laparoscopic Skills
(GOALS) [8]. Still, there is a demand for more automatic
assessment methods, which are objective and demand less
Materials and methods
effort from experienced surgeons in the evaluation process.
An automatic assessment method requires a set of
performance metrics. A great deal of research has been
performed to identify the psychomotor skills in laparo-
The participants were divided into three groups on the basis
scopic surgery that discriminate between different levels
of their experience level, indicated in a questionnaire
of surgical proficiency [9]. Similar studies have also been
stating their experience and dominant hand. Experience
performed regarding training for robotic surgery [10, 11].
was measured by the number of laparoscopic procedures
Finding metrics that define the quality of a surgical
they had performed. Experts had performed 500 or more
performance is not merely useful in the validation of the
laparoscopic procedures; intermediates, 31500 proce-
effect of the training tool. It is also important for pro-
dures; and the participants without laparoscopic experience
viding objective feedback to the user during training on
were novices. The novices were recruited from medical
simulators. Whether the results presented by the simulator
students in the 5th and 6th year of study and interns in their
reflect the real quality of the performance, i.e., the sim-
1st year of practice after medical school.
ulator has construct validity, can be found by checking
whether the built-in assessment tools are able to distin-
guish the performance of surgeons with different levels of Task
experience [2]. When a training curriculum on a simu-
lator is developed and validated, the residents can train In order to measure the participants psychomotor skills,
on the simulator until a proficiency-based level is they were asked to perform a task in a box trainer: D-box
reached. This takes into account the variations in the Basic simulator (D-Box Medical, Lier, Norway). A laby-
participants learning curves, in contrast to traditional rinth exercise was performed where they had to maneuver
time-based training [12]. The simulator has the potential and thread a needle through a loop of 10 holes (Fig. 1). The
to indicate when a resident is ready for the next level of needle was controlled with laparoscopic graspers (Endo
training, and tests on the simulators can be used in a Clinch II, AutoSuture, Covidien, Mansfield, MA, USA),
certification scheme to determine when the resident has one in each hand. The position of the tip of the two
attained the psychomotor skills required to perform sur- instruments was followed throughout the exercise using
gery on patients. Looking further into the future, moni- electromagnetic position tracking. The time to complete
toring the motions of, e.g., laparoscopic instruments in the exercise was measured as well. A camera was placed
the operating room could provide improved patient safety inside the box trainer, and the participants navigated the
by indicating potential harmful adverse events before instruments by viewing the images displayed on a screen
their occurrence. placed in front of them. The camera remained in a fixed
The goal of this study was to investigate a set of motion- position during the test so that it showed the entire
related metrics for their relevance to assess psychomotor labyrinth.
skills in MIS. Most of the metrics have previously been Before starting the test, each participant was briefly
presented elsewhere [9]. However, the formula for calcu- instructed on how to perform the task. All participants
lation bimanual dexterity was defined in this study. Nev- performed the test twice, where the first attempt was con-
ertheless, the novelty of the study is not limited to sidered a warm-up. The results from the second trial were
identifying performance measures that have not previously used to study the differences in performance between
been published. Every training setup and tasks need to be experts, intermediates, and novices.

856 Surg Endosc (2013) 27:854863

Fig. 1 Experimental setup. The box trainer with the labyrinth

exercise seen on the monitor, and the electromagnetic field generator
located on the left side of the box trainer (photo by Pall Jens

Position tracking

The position of the instruments were tracked with the Fig. 2 Definition of the orientation of the laparoscopic grasper
Aurora Electromagnetic Measurement System (Northern
Digital Inc., Waterloo, ON, Canada). The system consists axis (Fig. 2). The following nine metrics were used in the
of a system control unit, maximum four system interface evaluation of the performance of the participants:
units for position sensor inputs, and a field generator that (1) (1) Time (T) Time from start to completion of the
generates an electromagnetic tracking volume. task, measured in seconds.
A positioning sensor with six degrees of freedom was (2) Bimanual dexterity (BD) The participants ability to
placed close to the tip of the laparoscopic instruments and a control two instruments at the same time. BD is found
reference sensor was placed inside the box trainer. The by calculating the correlation between the velocity of
sensors measured positions with an accuracy of 0.48 mm, the tip of the instruments controlled by the left and the
according to the manufacturer. The positions were sampled right hand:
at a frequency of 40 Hz. PN   
n1 vleft n  v left vright n  vright
BD q
PN   2 PN  2
Evaluation metrics
n1 vleft n  v left n1 vright n  v right

The acquired motion data were analyzed by in-house where v is the velocity of the instruments and v denotes
developed motion analysis software. The software was the average velocity over the duration of the exercise.
written in Matlab 7 (MathWorks, MA, USA). (3) Path length (PL) Total movement of the tip of the
In order to eliminate high-frequency background noise, instrument for the duration of the task, measured in
the motion data were filtered with a sixth-order low-pass meters:
Butterworth filter with cutoff frequency at 6 Hz. s
All motion-related metrics were derived from the posi- ZT  2  2  2
dx dy dz
tion and the orientation of the instruments. The position PL dt
dt dt dt
was defined by xt; yt; ztTt0 . The orientation was 0

defined by at; bt; ctTt0 , where a and b measure the (4) Angular length (AL) The total change in the angle of
orientation in two planes perpendicular to the instruments the tip of the instrument in the plane perpendicular to
axis and c measures the rotation around the instruments the instruments axis, measured in degrees:

Surg Endosc (2013) 27:854863 857


  2 Results
da 2 db
AL dt
dt dt A total of 32 participants were included in the study: 7
experts, 14 intermediates and 11 novices. Five participants
(5) Depth perception (DP) DP is calculated by the total were women (1 intermediate and 4 novices), and 3 were
distance traveled by the tip of the instrument in the left-handed (1 intermediate and 2 novices).
instruments axis direction, measured in meters: Figure 3 shows box plots of the results from the three
groups for all the nine parameters included in the motion
ZT s
dz analysis. Separate values for the instrument controlled by
DP  dt
 dt  the dominant and the nondominant hand are presented for
0 seven of the parameters. Time and bimanual dexterity have
(6) Response orientation (RO) The total amount of the same value for both hands.
instrument rotation around its axis, measured in The P-values comparing the results of each pair of the
degrees: three groups of participants are presented in Table 1. Sta-
tistical significant differences were obtained between
ZT s experts and novices (EN in Table 1) for seven metrics:
RO  dt time, bimanual dexterity, path length, angular length, depth
perception, response orientation, and number of submov-
ements. Between the intermediates and novices (IN in
(7) Motion smoothness (MS) The total change in Table 1), significance was found for six metrics: time, path
acceleration of the tip of the instrument. The motion length, angular length, depth perception, response orien-
smoothness is measured in m/s3, and is normalized by tation (only for the dominant hand), and number of sub-
the duration of the task: movements. The results of the experts and intermediates
v (EI in Table 1) show that there are three parameters with
u ZT
u significant differences: bimanual dexterity, path length, and
J u1 d3 x
MS ; J t j2 dt; j 3 depth perception. For path length and depth perception,
T 2 dt
0 significance was found only for the nondominant hand.
For bimanual dexterity, the intermediates and the nov-
(8) Number of submovements (NoS) A submovement is ices performed at approximately the same level, as shown
defined by a movement of the tip of the instrument in Fig. 3, while the experts were superior in simultaneous
containing a velocity peak of at least 15 mm/s. use of both hands. This is in contrast to most of the other
(9) Average velocity (AV) The average velocity of the tip parameters, where both the expert and the intermediates
of the instrument for the duration of the task, outperformed the novices.
measured in mm/s. The novices scored significantly lower values for the
Bimanual dexterity is a subjective metric of the nondominant hand for the motion smoothness compared to
GOALS [8] assessment method. To make it more the experts and intermediates. No statistically significant
objective, the formula for calculating the bimanual dex- difference between the three groups was found for the
terity from motion data was defined in this study. The average velocity.
rest of the metrics have previously been used in other The correlation between the metrics identified as mea-
studies [9]. Considerations were taken in selecting task- suring statistical significant differences is shown in
independent metrics, which can be calculated for any Table 2. Time, path length, angular length, depth percep-
laparoscopic training module. tion, and number of submovements are found to be inter-
correlated and are thus partially dependent. Response
orientation and bimanual dexterity have less dependency to
Statistical analysis the other metrics and to each other.

The statistical analysis was performed by SPSS software,

version 19 (SPSS Inc., Chicago, IL, USA). The Mann Discussion
Whitney U-test was performed, and a probability of
P \ 0.05 was considered statistically significant. In order Our goal was to find a set of performance metrics by means
to find correlation between the metrics, Pearsons test was of motion analysis of the laparoscopic instrumentsmet-
applied to the metrics in which statistically significance rics that can be used to objectively assess psychomotor
was identified with the MannWhitney U-test. skills on the basis of surgical proficiency. A total of 32

858 Surg Endosc (2013) 27:854863




Fig. 3 AP Results of the labyrinth test presented as box plots of the whiskers show the fifth and 95th percentile. Outliers are plotted with
9 metrics. The middle band shows the median value; the bottom and circles and extreme outliers are marked with asterisks
top of the boxes show the 25th and 75th percentile; and the ends of the

participants were included in the study, divided into three psychomotor skills [14]. The intermediates, who had per-
groups: experts (n = 7), intermediates (n = 14), and nov- formed up to 500 procedures, might already have reached
ices (11). the highest level of skills. However, the lack of an external
The participants proficiency level was determined by objective assessment of the skills of each of the participants
self-assessment of the number of laparoscopic procedures left us with the number of procedures performed as the
they had performed. One might argue that self-judgment of indication of competence level [15].
competence can be unreliable [13], and that having per- Statistically significant differences in performance were
formed more procedures does not necessarily imply higher found for seven of the nine metrics when comparing the

Surg Endosc (2013) 27:854863 859




Fig. 3 continued

group of experts to the group of novices, six when com- movement perpendicular to the instruments axis (angular
paring intermediates to novices, and three when comparing length), and the total rotation of the instruments (response
experts to intermediates. The participants with laparoscopic orientation). These results are in accordance with other
experience (experts and intermediates) were superior to the studies of similar tasks, i.e., transfer of objects with lapa-
novices considering the time to complete the task and the roscopic instruments, performed either by using box
instrument movement metrics; the total traveled distance of trainers or VR simulators. Pellen et al. [16] found construct
the instruments (path length), the number of submov- validity for the time and path length metrics for an object-
ements of the instruments, the total traveled distance along positioning task on the ProMIS simulator. Woodrum et al.
the instruments axis (depth perception), the amount of [17], in addition to time and path length, found construct

860 Surg Endosc (2013) 27:854863



Fig. 3 continued

Table 1 Results of the descriptive statistics on the metrics of the three study groups (P-values)

Time 0.2 0.001* 0.001*

Bimanual dexterity 0.01* 0.1 0.02*
Dominant hand Nondominant hand

Path length 0.09 0.001* 0.001* 0.04* 0.001* 0.001*

Angular length 0.4 0.002* 0.002* 0.3 0.03* 0.004*
Depth perception 0.2 0.001* 0.001* 0.04* 0.001* 0.001*
Response orientation 0.7 0.006* 0.01* 0.3 0.1 0.04*
Motion smoothness 0.7 0.07 0.06 0.5 0.03*a 0.02*a
Number of submovements 0.2 0.002* 0.002* 0.2 0.001* 0.001*
Average velocity 0.8 0.6 0.7 0.9 0.9 0.7

E expert, I intermediate, N novice

* Significant at P \ 0.05
Statistical significance was found, but the group with the lowest values was the opposite of the expected result

validity for the angular length metric (right instrument significant differences in the performance on tests per-
only) for the Lifting and Grasping module on the Lap- formed in a box trainer for time, path length, and depth
Sim VR simulator. Chmarra et al. [15] measured statistical perception.

Surg Endosc (2013) 27:854863 861

Table 2 Correlation between the metrics identified as measuring statistically significant differences
Hand Time Bimanual Path Angular Depth Response Number of
dexterity length length perception orientation submovements

Dominant hand
Time 100 44 95 92 94 63 97
Bimanual dexterity 44 100 38 30 38 11 38
Path length 95 38 100 92 100 67 99
Angular length 92 30 92 100 90 62 93
Depth perception 94 38 100 90 100 68 99
Response orientation 63 11 67 62 68 100 68
Number of submovements 97 38 99 93 99 68 100
Nondominant hand
Time 100 44 95 72 95 43 98
Bimanual dexterity 44 100 43 42 45 38 40
Path length 95 43 100 81 100 49 95
Angular length 72 42 81 100 82 53 67
Depth perception 95 45 100 82 100 49 95
Response orientation 43 38 49 53 49 100 36
Number of submovements 98 40 95 67 95 36 100
Pearsons test was used to calculate the correlation matrix for the metrics identified as statistically significant in Table 1. The correlations are
calculated for the dominant (upper) and the nondominant (lower) hand separately

Statistically significant differences were found for three The main purpose of assessing a surgeons psychomotor
metrics when comparing experts to intermediates: bimanual skills is to assure the quality of surgical performance by
dexterity, path length, and response orientation. For the latter quantifying performance levels, which can be used to
two metrics, significance was found only for the nondomi- evaluate the effect of training or objectively assess dex-
nant hand. The bimanual dexterity measures the ability to terity as part of a certification test [9]. Even though
control both instruments simultaneously, which requires a bimanual dexterity separates the performance of the most
high level of control of the nondominant hand, as well as the experienced laparoscopic surgeons from the ones with less
dominant hand. This ability appears not to be fully devel- experience, care should be taken in purely assessing psy-
oped among the participants in the intermediate group. The chomotor skills on the basis of the bimanual dexterity
intermediates are moving the instrument in the nondominant metric. It would be easy to improve the score of the metric
hand more than the experts, but they have less simultaneous, by moving both hands at the same time, without improving
coordinated movements. The results of this study suggest the efficiency or outcome of the procedure. Nevertheless, it
that the control of the nondominant hand and the higher might be beneficial for the laparoscopic surgeons who have
degree of simultaneous hand movements are metrics that reached a certain level of dexterity to focus more of their
differentiate between surgeons who have performed less psychomotor skills training on the nondominant hand, i.e.,
than 500 laparoscopic procedures from those who have by the use of simulators with modules that exercise the
performed more than 500 procedures. The skill of simulta- control of both hands, including simultaneous hand
neous and coordinated hand movement appears not to be movements.
fully acquired until a later stage in the learning process, In general, it is expected that the more experienced
which is supported by the fact that the intermediates did not surgeons have smoother motions and thus obtain lower
outperform the novices in bimanual dexterity. values for the motion smoothness metric, compared to the
By investigating the correlation of the results, most of less experienced ones. However, the results showed the
the metrics were found to be strongly dependent. However, opposite; the group of novices attained the lowest motion
bimanual dexterity (and response orientation) had a lower smoothness. This could be related to the nature of the
correlation to other metrics. This supports the fact that exercise, where the participants are motivated to pass the
bimanual dexterity is measuring a different aspect of psy- needle through the labyrinth of holes in the most efficient
chomotor skills than the traditionally parameters like time manner possible, and the exercise required larger move-
and path length. ments compared to normal surgery. The more experienced

862 Surg Endosc (2013) 27:854863

surgeons performed the exercise the quickest and were thus Conclusions
likely to have moved the instruments between the holes in a
rapid manner that increased the motion smoothness value. Motion analysis is a valid objective way of assessing
Because no penalty points were introduced for committed psychomotor MIS skills. Experts and intermediates out-
errors (e.g., bumping the instruments into obstacles or performed novices for six out of nine tested metrics. The
dropping the needle), we expect that all the participants difference between experts and intermediate was smaller;
tried to perform the task as fast as possible. Introducing separation between the two groups was found for three
penalty points might influence performance of the task metrics. An experienced MIS surgeon can be distinguished
because the participants would have to take into account from a less experienced one by the higher ability to control
the preferred manner of performing the task. Including the instrument in the nondominant hand and the higher
penalty points would require an observer evaluating the degree of simultaneous (coordinated) movements of the
performance, either during the task or by video assessment. two instruments. This finding indicates that less experi-
Error scores are, however, common on VR simulators, enced surgeons could benefit from training nondominant
where they are automatically detected by the computer hand dexterity and bimanual dexterity.
[18]. Our result is similar to the findings of Chmarra et al.
[19] and Maithel et al. [20], neither of whom was able to Acknowledgments The authors thank the surgeons, residents,
medical students, and interns for participating in this study.
separate the groups of different levels of experience by the
motion smoothness metric. Disclosures Erlend F. Hofstad, Cecilie Vapenstad, Magdalena K.
No statistically significant difference was found between Chmarra, Thomas Lang, Esther Kuhry, and Ronald Marvik have no
the groups for the average velocity of the tip of the conflicts of interests or financial ties to disclose.
instruments. This is consistent with the results found for the
time and path length. The novices needed more time to References
complete the task and moved the instruments more than the
experts and intermediates. The more experienced surgeons 1. Hallam DM, Anwar MA, Khan IM, Memon MA, Steele K, Lidor
moved the instruments in a more efficient manner, but A (2008) Access to the abdomen. In: Scott-Conner CEH (ed) The
those movements were not necessarily faster than those of SAGES manual of strategic decision making. Springer, New
York, pp 19
the inexperienced participants. 2. Feldman LS, Sherman V, Fried GM (2004) Using simulators to
To establish a passfail rate in a certification scheme, assess laparoscopic competence: ready for widespread use? Sur-
it might not always be necessary to know what dis- gery 135(1):2842
criminates between a surgeon at an expert level and the 3. Subramonian K, DeSylva S, Bishai P, Thompson P, Muir G
(2004) Acquiring surgical skills: a comparative study of open
one at an intermediate level, as they both normally have versus laparoscopic surgery. Eur Urol 45(3):346351
skills sufficient to perform surgery safely. It is, however, 4. Gurusamy KS, Aggarwal R, Palanivelu L, Davidson BR (2009)
important to be able to recognize when residents are Virtual reality training for surgical trainees in laparoscopic sur-
ready to move to the next training level, or which sur- gery. Cochrane Database Syst Rev (1):CD006575
5. Basdogan C, Sedef M, Harders M, Wesarg S (2007) VR-based
geons have superior skills that allow them to perform simulators for training in minimally invasive surgery. IEEE
specific surgeries. Knowledge of the expert level of Comput Graph Appl 27(2):5466
performance is important when evaluating training cur- 6. Carter FJ, Schijven MP, Aggarwal R, Grantcharov T, Francis NK,
ricula. This knowledge is also essential in the validation Hanna GB, Jakimowicz JJ (2005) Consensus guidelines for val-
idation of virtual reality surgical simulators. Surg Endosc 19(12):
process of simulators. When determining construct 15231532
validity, the built-in assessment metrics are tested in 7. Gallagher AG, Ritter EM, Satava RM (2003) Fundamental
order to find out if they are appropriate to measure dif- principles of validation, and reliability: rigorous science for the
ferences in performance level. For instance, one can assessment of surgical education and training. Surg Endosc
assess whether the most experienced surgeons outperform 8. Vassiliou MC, Feldman LS, Andrew CG, Bergman S, Leffondre K,
the less experienced ones by comparing results provided Stanbridge D, Fried GM (2005) A global assessment tool for evalu-
by the simulator. Because it is essential to ensure that ation of intraoperative laparoscopic skills. Am J Surg 190(1):107113
correct feedback is given to the ones who train on the 9. Oropesa I, Sanchez-Gonzalez P, Lamata P, Chmarra MK, Pag-
ador JB, Sanchez-Margallo JA, Sanchez-Margallo FM, Gomez EJ
simulator, it is necessary to further investigate what that (2011) Methods and tools for objective assessment of psycho-
feedback should contain and how it should be presented motor skills in laparoscopic surgery. J Surg Res 171(1):e81e95
to them. Ideally, once construct validity of the training 10. Narazaki K, Oleynikov D, Stergiou N (2006) Robotic surgery
setup is established, indication on the skills needed to be training and performance: identifying objective variables for
quantifying the extent of proficiency. Surg Endosc 20(1):96103
improved and a list of suitable training modules to 11. Judkins TN, Oleynikov D, Stergiou N (2009) Objective evalua-
improve those skills should be presented after a training tion of expert and novice performance during robotic surgical
session. training tasks. Surg Endosc 23(3):590597

Surg Endosc (2013) 27:854863 863

12. Satava RM (2008) Historical review of surgical simulationa 17. Woodrum DT, Andreatta PB, Yellamanchilli RK, Feryus L,
personal perspective. World J Surg 32(2):141148 Gauger PG, Minter RM (2006) Construct validity of the LapSim
13. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe laparoscopic surgical simulator. Am J Surg 191(1):2832
KE, Perrier L (2006) Accuracy of physician self-assessment 18. Thijssen AS, Schijven MP (2010) Contemporary virtual reality
compared with observed measures of competence: a systematic laparoscopy simulators: quicksand or solid grounds for assessing
review. JAMA 296(9):10941102 surgical trainees? Am J Surg 199(4):529541
14. Schout BM, Hendrikx AJ, Scheele F, Bemelmans BL, Scherpbier AJ 19. Chmarra MK, Kolkman W, Jansen FW, Grimbergen CA, Dank-
(2010) Validation and implementation of surgical simulators: a crit- elman J (2007) The influence of experience and camera holding
ical review of present, past, and future. Surg Endosc 24(3):536546 on laparoscopic instrument movements measured with the
15. Chmarra KM, Klein S, de Winter F, Jansen JC, Dankelman FW et al TrEndo tracking system. Surg Endosc 21(11):20692075
(2010) Objective classification of residents based on their psy- 20. Maithel SK, Villegas L, Stylopoulos N, Dawson S, Jones DB
chomotor laparoscopic skills. Volume 24. Springer, Heidelberg (2005) Simulated laparoscopy using a head-mounted display vs
16. Pellen MG, Horgan LF, Barton JR, Attwood SE (2009) Construct traditional video monitor: an assessment of performance and
validity of the ProMIS laparoscopic simulator. Surg Endosc muscle fatigue. Surg Endosc 19(3):406411