DaVinciCaseStudy

ANOVA case study: the Da Vinci system
Dr Alberto Corrias
Department of Biomedical Engineering. National University of Singapore
BN2102 Bioengineering Data Analysis ANOVA case study: the Da Vinci system 1 / 39
PollEv game Standings
Standings after week 2 and week 3

Lucky draw will happen at the start
of the class
Next lucky draw in two weeks with
all students reset to zero, but overall
cumulative standing will still be
calculated and published for the final
prize.
Laparoscopy versus open surgery
The Da Vinci system
Youtube link: https://www.youtube.com/watch?v=VJ_3GJNz4fgw
The Da Vinci system - doctor’s opinion
Youtube link: https://www.youtube.com/watch?v=DLj4ImsVkDQ
Statistics of the Da Vinci system
Approved for
Totally Endoscopic 64
Atrial Septal Defect (ASD) countries where
Gynecological Laparoscopic Procedures system is in use
Thoracoscopically-Assisted
Cardiotomy Procedures
Mitral valve repair surgery
General Laparoscopic Surgery
∼10K
publications on
(gallbladder, gastroesophageal reflux and
the DaVinci
gynecologic surgery)
system since 1998
Laparoscopic Radical Prostatectomy
†
Sources:
Intuitive Surgical Website: http://www.intuitivesurgical.com/
FDA website: www.fda.gov
Our first scenario
You are employed by the Intuitive

Surgical training center, in charge
of the da Vinci Skills Simulator.
You recruit 40 medical students,
divide them in 4 groups, each
undergoing a different training
protocol.
The main issue
You want to know whether there is
any difference among the proposed
training protocols.
Our fist scenario: the 4 protocols
Protocol G1 (cone game) Protocol G2 (rings game)
Protocol G3 (bowls) Protocol S (suture)
†
Video link http://bcove.me/pi91olz5
Our first scenario: training 40 students
Our first scenario: evaluation of performance
Participants evaluated using the Global Evaluative Assessment of
Robotic Skills (GEARS) on an in vivo operation on a pig.
The data
GEARS Scores (%)

Protocol Protocol Protocol Protocol
G1 G2 G3 S
50.90 26.69 44.12 50.34
33.87 35.08 37.85 57.48
27.35 33.92 48.49 67.45
14.86 38.18 33.48 71.10 We have m = 4 groups
50.32 46.38 24.80 55.12 n = 10 for all groups
41.34 62.75 53.52 36.62
44.73 24.75 38.74 66.81
23.29 70.92 37.68 64.74
25.71 47.03 43.31 52.66
40.12 19.29 42.49 54.98
The problem in statistical terms
What we want to know is whether all the participants are able to

achieve the same scores regardless of their training protocol. If that’s
the case, all the sample scores we obtained would have been drawn
from one single population (or 4 identical ones). If not, it means the
4 samples were drawn from 2 or more population, i.e., populations
with different means.
Hypothesis
H0 : All scores drawn from one population (informal way
”Training protocol does not matter”)
H1 : Scores were drawn from different populations (informal way
”Training protocol matters”)
Looking at the data using boxplots
Looking at the data using histograms
Performing ANOVA calculations
1 Calculate the sample means

Average Scores (X )
G1 G2 G3 S
35.25 40.50 40.45 57.73
Pn 2
i=1 (Xi −X )
2 Calculate sample variances s 2 = n−1
Sample Variances (s 2 )
G1 G2 G3 S
148.54 273.82 63.57 105.02
Performing ANOVA calculations
3 Calculate the mean of the average scores
X G1 + X G2 + X G3 + X S
XX = = 43.4815
4
4 Calculate (sX2 )
(X G 1 − X X )2 + (X G 2 − X X )2 + ...
sX2 = = 96.297
4−1
5 2 and s 2
Compute key quantities swit bet
2
sbet = nsX2 = 10(96.297) = 962.97
2 sG2 1 + sG2 2 + sG2 3 + sS2

swit = = 147.74
4
More on the role of the sample size
You are analyzing data from 4 groups, each with n = 10 subjects. You have
2 2
computed swit and sbet . At this point you are told that actually the experiment was
run on 20 subjects per group and they give you the additional data that was
missing. Now n = 20 per group. Assuming that all the additional measurements
Xi just happened to be exactly of the same value as the sample means X you had
computed initially,which of the following is true about the new analysis with
n = 20
2 2
1 swit will become bigger, but sbet will become smaller
2 2
2 swit will become smaller, but sbet will become bigger
2 2
3 swit and sbet will become bigger
2 2
4 swit and sbet will become smaller
ANSWER:
Pn 2
2 2 i=1 (Xi −X )
swit will be smaller. swit include terms like n−1
2 2
sbet will be bigger. sbet = nsX2
Performing the actual test at α = 0.05 significance
0.6 F distribution, F (3, 36)
2
sbet
F = 2
swit
= 6.518 This light blue
0.4
Numerator degrees area is 0.05
of freedm:
m−1=3
0.2
Denominator
degrees of freedom:
m(n − 1) = 36
fcrit
No Rejection Region Rejection Region
The F distribution
Suppose you are doing 2 ANOVA analysis
ANOVA A: m = 3 groups with n = 4 subjects per group
ANOVA B: m = 3 groups with n = 40 subjects per group
Both analysis are conducted using the same 95% significance. For ANOVA A, you
A B
compute fcrit , while for ANOVA B, you compute fcrit . Intuitively, which is true?
A B
1 fcrit > fcrit
A
2 fcrit B
< fcrit
A
3 fcrit B
= fcrit
A B
ANSWER: fcrit > fcrit . Intuitively, the smaller n, the more uncertainty, the larger
the tail area, the greater fcrit will be for the same area.
0.8
0.6 A : ν1 = 2, ν2 = 9
0.4 B : ν1 = 2, ν2 = 127
0.2
0
0 1 2 3 4
Determining fcrit using tables
This purple area is We reject the NULL

0.4
the p value hypothesis as our
computed F ratio is
> fcrit . We already
0.2 know that the p value is
< 0.05†
∼ 2.9 6.518

.
†
Exact p value can be computed by statistical software
Rejecting the NULL hypothesis: interpretation
In statistical terms
The differences in scores among the 4 groups are statistically
significant (at 95% level). This means that, if all groups were drawn
from a single population, the probability of obtaining samples
displaying the observed differences are less than 5%. We still don’t
know which specific group(s) is (are) responsible for the observed
differences.
How it is reported
We observed statistically significant differences (p < 0.05) in
performance scores among the 4 groups who underwent different
training protocols.
Statistical interpretations
Our analysis found F = 6.518 which is greater than the Fcrit = 2.9.
Imagine, for a moment, that, instead, the calculations ended up in
computing F = 1.9. In such a case, can we conclude that there is no
difference among the 4 protocols? ANSWER: NO! ANOVA is still a
form of hypothesis testing. When we fail to reject the NULL
hypothesis, we can’t conclude that the NULL hypothesis is true. All
we can say is that the data failed to support any difference among the
protocol. The hypothesis that the training protocol does not matter is
still plausible (NOTE: ”plausible”, NOT ”true”!)
Pairwise t-tests: the Bonferroni method
Each t test will have
X1 − X2
Six possible pairwise Student t t=p 2 2
swit /n1 + swit /n2
tests
1 G1 vs G2 The Student t distribution
2 G1 vs G3 will have m(n − 1) degrees of
3 G1 vs S P (N − m where
freedom
4 G2 vs G3 N = ni , if groups have
different sizes)
5 G2 vs S
If we choose a significance
6 G3 vs S level of 95%, then we will use
a tail area of 0.05
6 = 0.0083 to
compute the tcrit value
The 6 Bonferroni t tests (e.g., 95%)
H0 : µ1 = µ2 ,H1 : µ1 6= µ2
G1vsG2 G1vsG3 G1vsS G2vsG3 G2vsS G3vsS
tstat -0.97 -0.96 -4.14 0.01 -3.17 -3.18
BONF
tcrit 2.79 2.79 2.79 2.79 2.79 2.79
tcrit 2.028 2.028 2.028 2.028 2.028 2.028
Reject? NO NO YES NO YES YES
p 0.34 0.35 0.0002 0.99 0.003 0.003
Original significance
value. Area = 0.05/2
When doing multiple
New significance value. comparisons after ANOVA,
Area = 0.0083/2 you should use tcrit BONF , not
the original tcrit !

tcrit BONF
tcrit
†
The figure above is a zoom-in on the right tail of the t distribution. The same happens on the left tail (not
shown). Remember that Bonferroni t tests are two-tailed.
The Bonferroni method
Suppose we have performed ANOVA with m = 3 groups. We are
interested in a 94% significance level. In this case, the area to the
BONF is?
right of tcrit
1 0.12
2 0.06
3 0.05
4 0.04
5 0.03
6 0.02
7 0.01
8 0.005
ANSWER: 0.01. The original α = 0.06. There are 3 comparisons.

BONF is
Hence, the total tail area is 0.02. The area to the right of tcrit
half of that, hence 0.01.
How results are reported
ANOVA analysis revealed significant differences among the 4 training protocols

(p < 0.05). Post − hoc analysis (Bonferroni method) revealed statistically
significant differences between protocol S and each of the other protocols (*
p = 0.003, ** p = 0.003, *** p = 0.0002). Data are unable to support any
difference among protocols G1, G2 and G3.
How results are reported: examples from BME papers
The ANOVA table
n((X G 1 − X X )2 + (X G 2 − X X )2 + (X G 3 − X X )2 + (X S − X X )2 )
2
sbet
2
swit
P G1
(Xi − X G 1 )2 + (XiG 2 − X G 2 )2 + (XiG 3 − X G 3 )2 + (XiS − X S )2
P P P
SS: sum of squares, df: degrees of freedom, MS: mean squares
Our second scenario: a more realistic exercise
Link to video
http://intuitivesurgical.com/products/skills_simulator/
Calculation question (3 points)
GEARS scores Question:
Medicine Urology Junior Use ANOVA to determine
Students Interns Doctors whether there is any
65.14 70.22 74.86 difference in GEARS scores
39.70 75.18 57.78 among the 3 groups. Provide
the values of F , Fcrit and your
43.68 66.30 65.17
answer using a 95%
72.17 73.53 66.29 confidence level.
Solution
1 Calculate the sample means

Average Scores (X )
Medicine Urology Junior
Students Interns Doctors
55.17 71.31 66.03
Pn 2
i=1 (Xi −X )
2 Calculate sample variances s 2 = n−1
Sample Variances (s 2 )
Medicine Urology Junior
Students Interns Doctors
253.25 15.40 48.95
Solution
3 Calculate the mean of the average scores
X students + X interns + X junior

XX = = 64.168
3
4 Calculate (sX2 )
(X student − X X )2 + (X intern − X X )2 + ...

sX2 = = 67.67
3−1
5 2 and s 2
Compute key quantities swit bet
2
sbet = nsX2 = 4(67.67) = 270.68
2
sstudent 2
+ sintern 2
+ sjunior
2
swit = = 105.864
3
Solution

2
sbet
F = 2
swit
= 2.557
0.6
Numerator degrees
of freedom: This light blue
m−1=2 0.4
area is 0.05
Denominator
degrees of freedom: 0.2
m(n − 1) = 9
fcrit
Determining fcrit using tables

From the table,
0.6 fcrit = 4.26. We fail to
reject the NULL
This purple area is hypothesis as our
0.4
the p value computed F ratio is
< fcrit . We already
0.2 know that the p value is
> 0.05.
2.557 4.26
ANOVA: assumptions check
Assumptions (similar to t test)

Truly independent and random samples
Look out for trends in the data. Measurements must be
independent from each other
Any bias in selecting subjects?
Underlying populations approximately normal
All samples drawn from population with approximately the same
variance (even if with different means)
FAQ: why do we fail to reject the NULL hypothesis when
F ∼ 1?
2
sbet
F = 2
swit
2 and s 2 are estimates of the unknown σ 2 of the

Both sbet wit
underlying population.
2 is simply the average of the sample variances. It is
Note that swit
unaffected by whether or not the means are different from each
other
On the other hand, the more one or more sample means are
2 will be!
”far” from the means of the means (X X ), the bigger sbet
2 (X G 1 − X X )2 + (X G 2 − X X )2 + ...
sbet = n(sX2 ) = n( )
m−1
Different types of ANOVA
Our case study today

So far, we looked at a study where one factor (the training protocol)
was hypothesized to influence the results (GEARS scores). Because of
this, the type of ANOVA we saw toady is referred to as one way
ANOVA.
Beyond one way ANOVA (not covered in BN2102)

If you need to analyze data from experiments where two factors are
hypothesized to influence the results, then the technique is called two
way ANOVA. In general, one can perform n way ANOVA when n
factors are believed to influence the results. The underlying principles
are similar to what we saw today.

DaVinciCaseStudy

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

DaVinciCaseStudy

Uploaded by

Copyright:

Available Formats

ANOVA case study: the Da Vinci system

Department of Biomedical Engineering. National University of Singapore

Standings after week 2 and week 3

Youtube link: https://www.youtube.com/watch?v=VJ_3GJNz4fgw

Youtube link: https://www.youtube.com/watch?v=DLj4ImsVkDQ

You are employed by the Intuitive

Protocol G3 (bowls) Protocol S (suture)

GEARS Scores (%)

What we want to know is whether all the participants are able to

1 Calculate the sample means

3 Calculate the mean of the average scores

2 sG2 1 + sG2 2 + sG2 3 + sS2

0.6 F distribution, F (3, 36)

No Rejection Region Rejection Region

0.6 F distribution, F (3, 36)

This purple area is We reject the NULL

No Rejection Region Rejection Region

Each t test will have

the original tcrit !

ANSWER: 0.01. The original α = 0.06. There are 3 comparisons.

ANOVA analysis revealed significant differences among the 4 training protocols

SS: sum of squares, df: degrees of freedom, MS: mean squares

1 Calculate the sample means

3 Calculate the mean of the average scores

X students + X interns + X junior

(X student − X X )2 + (X intern − X X )2 + ...

0.8 F distribution, F (2, 9)

0.8 F distribution, F (2, 9)

Assumptions (similar to t test)

2 and s 2 are estimates of the unknown σ 2 of the

Our case study today

Beyond one way ANOVA (not covered in BN2102)

You might also like