An Empirical Analysis of Vendor

Response to Vulnerability Disclosure

Ashish Arora, Ramayya Krishnan, Rahul Telang, Yubao

Yang

Carnegie Mellon University

1
Motivation

• Information security 160,000

breaches: A significant and 140,000

increasing threat 120,000

100,000
137,529

80,000

82,094
60,000

• Lack of systematic policy for 40,000

52,658

21,756

how vulnerability information 20,000

252 406 773 1,334 2,340 2,412 2,573 2,134
9,859
3,734

should be disclosed 0
1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003

Self reported security incidences

2
Motivation

• While theoretical models are useful to understand the

issues surrounding vulnerability disclosure, we need
empirical estimates for policy making.

• One of the key factors is to understand how vendors

respond to disclosure and disclosure policies?

• An empirical estimate on vendor response to

disclosure window will be very useful in calibrating
the current policies. However, data collection is non-
trivial.
3
Research goals

• Whether (and by how much) early disclosure induces

vendors to patch faster.

• What are other key factors that condition patching

time?

4
Literature
• Arora, Telang, and Xu (2003) outline a model for the optimal
policy for software vulnerability disclosure.

• Telang and Wattal (2004) show that disclosure is costly to

vendors and hence provides incentives to vendors to improve
the quality of their software

• Market based mechanism

– Camp and Wolfram (2004) describe a means for creating market for
vulnerabilities in order to increase the security of systems
– Kannan and Telang (2004) show that markets always perform worse that
CERT because of poor disclosure rules
– Schechter (2002) argues that vendors should create and exploit a market
for testers
– Ozment (2004), an auction based market based mechanism

5


Predictions of Analytical Model (Arora, Telang and Xu

[2003])

• Vendors face cost of patching. More time they have for patching
less it costs them.
• Vendors’ customer incur loss when they are breached. Depending
on the market structure, vendors “internalize” some of the
customer loss. The more loss they internalize, more costs they
incur and earlier is the patch.
• Disclosure of vulnerability is potentially hurtful to customers
because disclosure makes it easier for hackers to find the
information too. Thus disclosure threat supposedly forces vendors
to patch faster because disclosure increases their costs.
• However, there is little (if any) empirical evidence that vendors
indeed patch faster and by how much.
6
Model Prediction

• Besides understanding the role of disclosure, we also

investigate other factors that have bearing on vendor
response. Some of the factors are
– Severity of the vulnerability
– Vendor characteristics
– Open source / closed source
– Disclosure source
– Publicly traded firm
– Effect of September 11.

7
Data
• Vulnerabilities published by SecurityFocus or CERT/CC.

• Information on the key time variables (Patching time = Date of

patch – Date of notification). CERT provided us with information
on when they notified the vendors. The date on which vendors
delivered a patch to them etc.

• Vendor information from Hoover’s online business information

database and vendor’s website

• Vulnerability information from the NIST ICAT database

• Time period from 9/26/2000 to 8/11/2003

• 1280 observations, related to 255 unique vendors and 303 unique

ICAT database documented vulnerabilities

8
CERT/CC Vs SecurityFocus

• Two major vulnerability disclosure sources

• CERT/CC (A Federal supported R&D center)
– Typically 45 days of secret period after notifying vendors
– No exploit code disclosed

• SecurityFocus (An online open forum)

– Policy of instant disclose (many time individuals may provide
vendors some time before disclosure)
– Disclose full information

• We discard all vulnerabilities which are reported first by vendors

9
Early disclosure
• Anytime vulnerability is disclosed within the disclosure window
(mostly 45 days) and vendor has not patched, early disclosure
happens. However, in our sample most of the time disclosure
happens quite early.
– Instant disclosure is a case when disclosure happens before or at
the same time when vendor is notified of the vulnerability.

• “Not early” case on SecurityFocus

– Identifiers tend to be careful in using this powerful instant disclosure
tool. They inform the vendor first and wait for the vendor patch
before posting on SecurityFocus website
– 30% in our sample
• “Early” case on CERT/CC
– Disclosure by others in CERT/CC secret time period
– Already known public when CERT/CC picked it up
– A vendor was missed when CERT/CC notify other vendors
– Disclosure before 45 days if 80% of the vendors are ready

10
Impact of instant disclosure
Without instant disclosure With instant disclosure
Patching Time (days) 58.08 (78.30) 44.37 (80.01)
Severity Metric 29.97 (22.68) 23.44 (21.34)
Obs / vuls 489 93 791 245

Impact of publication source

Published by CERT/CC Published by SecurityFocus
Patching 48.41 (78.13) 63.91 (94.79)
Severity Metric 27.38 (22.37) 8.76 (4.48)
Obs / vuls 1181 / 258 99 / 43

11
Impact of disclosure source (for Instantly
disclosed vuls)

CERT/CC SecurityFocus Others

Patching Time 24.46 (36.96) 42.95 (78.59) 59.42 (97.71)
Metric 38.41 (23.79) 21.60 (21.63) 16.97 (13.33)
Obs / vuls 153 24 398 147 240 74

• Disclosure by CERT has a significant impact on patching speed

of the vendor than disclosure by Securityfocus or by other
sources

12
Vendor Characteristics
Mean Std Dev
No. of Employee 22640 75997
Open Source 0.21 0.41
Public Firms 0.42 0.50

There are total 255 unique vendors. Above statistics is based on the 121 vendors
that we have reliable information.

Vulnerability Characteristics
• There are total 301 unique vulnerabilities.
• Average Severity Score was 16.25.
• Each vulnerability affected on an average 11 vendors.

13
Analysis
• Two sets of analysis
– Impact of disclosure on patching time.
• Conditional on not having patched until time t-1,
how will disclosure at time t will affect vendor’s
patching speed.
• We choose different values of t.
– Impact of expected “disclosure window” on
patching time.
• How will change in disclosure window affect
vendors’ patching behavior?
14
 Te = 0 days Te = 4 - 7 days
(1.1) Vendor (1.2) Vendor (2.1) Vendor (2.2) Vendor
fixed effect characteristics fixed effect characteristics
CERT -0.55 (0.18) -0.47 (0.17) 0.27 (0.21) 0.30 (0.18)
Disclosure -0.78 (0.10) -0.86 (0.10) -0.50 (0.20) -0.43 (0.20)
Firm Size 0.00 (0.01) 0.00 (0.02)
Public firm -0.06 (0.13) 0.08 (0.14)
Open source -0.59 (0.35) -0.52 (0.12) -1.05 (0.40) -0.20 (0.15)
Severity metric -0.08 (0.04) -0.07 (0.04) -0.04 (0.06) -0.06 (0.05)
Post
September/11 -0.44 (0.11) -0.41 (0.10) 0.08 (0.15) 0.08 (0.14)
Constant 4.44 (0.23) 4.37 (0.22) 3.81 (0.27) 3. 69 (0.25)
R2 0.0883 0.0849 0.0254 0.0205
N 1280 1280 388 388
Notes: * indicates significant at 10% level, ** indicates significant at 5% level and *** indicates significant at 1% level.

15
Results
• Disclosure accelerates the patch delivery significantly. For
vulnerabilities that are disclosed instantly, patch comes
55% faster than otherwise.

• When disclosure happens later the patch still comes

significantly faster but the difference between with and
without disclosure patching speed seems to reduce.

• Open source vendors tend to patch faster; almost 44%

faster.

• Significant impact of 9/11. Patches come faster post 9/11.

16
Impact of Disclosure Source
Te=0 days
(1.1) Vendor
(1.2) Vendor characteristics
fixed effect
C_C -1.02** (0.23) -0.95** (0.21)
C_S -1.01** (0.20) -1.06** (0.18)
C_O -0.63** (0.21) -0.60** (0.19)
C_None -0.04 (0.20) 0.04 (0.18)
Firm Size -0.55* (0.35) 0.00 (0.01)
Public firm -0.06 (0.13)
Open source -0.52** (0.12)
-0.07* (0.04) -0.07* (0.04)
Severity metric (log)

Post September/11 -0.41** (0.11) -0.38** (0.11)

Constant 3.92** (0.22) 3.88** (0.21)
R2 0.1012 0.0966

N 1280 1280
Notes: * indicates significant at 10% level, ** indicates significant at 5% level

17
 Te=0 days Te=4 - 7 days
(1.1) Vendor (1.2) Vendor (2.1) Vendor (2.2) Vendor
fixed effect characteristics fixed effect characteristics
-1.02 *** (0.23) -0.95 *** (0.21) -0.69 * (0.36) -0.65 * (0.37)
C_C
-1.01 *** (0.20) -1.06 *** (0.18) -0.54 (0.46) -0.38 (0.46)
C_S
-0.63 *** (0.21) -0.60 *** (0.19) 2.04 *** (0.74) 1.64 ** (0.70)
C_O
-0.04 (0.20) 0.04 (0.18) 0.52 *** (0.19) 0.47 *** (0.17)
C_None
0.00 (0.01) -0.01 (0.02)
Firm Size
-0.06 (0.13) 0.07 (0.13)
Public firm
-0.55 * (0.35) -0.52 *** (0.12) -1.05 *** (0.39) -0.17 (0.15)
Open source
-0.07 * (0.04) -0.07 * (0.04) -0.05 (0.05) -0.06 (0.05)
Severity metric
(log)
-0.41 *** (0.11) -0.38 *** (0.11) 0.12 (0.15) 0.09 (0.14)
Post September/11
3.92 *** (0.22) 3.88 *** (0.21) 3.56 *** (0.25) 3.56 *** (0.24)
Constant
0.1012 0.0966 0.0486 0.0482
R2
N 1280 1280 388 388
Notes: * indicates significant at 10% level, ** indicates significant at 5% level and *** indicates significant at 1% level.

18
Impact of Disclosure Window “T”
• We now want to understand what is the impact of disclosure
window on patching time. This is the information a policy maker
like CERT needs. Before they decide how much time should be
given vendors, they need to know what is impact of giving one
additional day.

• CERT provides approximately 45 days. However, it is clear the

most of the time disclosure happens much earlier. This means
that expected disclosure window “T” is much smaller and is
unobservable to econometrician.

• But we know that for all vulnerabilities that are disclosed

instantly, T = 0. For all other, T>0. Thus these two samples
should provide us with the directional effect of “T” on patching
time.

19
Impact of disclosure window “T”

• We use only CERT data to analyze this

because CERT has a more well defined
policy.
• We test whether there is significant
difference between patching times for
vulns instantly disclosed and otherwise
in the CERT sample.

20
 With disclosure source Without disclosure source
(1.1) Vendor (1.2) Vendor (2.1)Vendor (2.2)Vendor
fixed effect characteristics fixed effect characteristics
Disclosure -0.83** (0.11) -0.93** (0.10)
Disclosed_by_C -0.97** (0.17) -0.99** (0.15)
Disclosed_by_S -0.94** (0.13) -1.09** (0.12)
Disclosed_by_O -0.56** (0.14) -0.63** (0.14)
Firm Size 0.00 (0.02) 0.00 (0.02)
Public firm -0.07 (0.14) -0.08 (0.14)
Open source -0.55* (0.36) -0.56** (0.13) -0.60* (0.36) -0.55** (0.13)
Severity metric -0.06 (0.04) -0.05 (0.04) -0.07* (0.04) -0.06* (0.04)
Post 9/11 -0.44** (0.12) -0.40** (0.11) -0.48** (0.12) -0.43** (0.11)
Constant 3.86** (0.19) 3.92** (0.19) 3.94** (0.18) 3.97** (0.19)
R2 0.0991 0.0953 0.0903 0.0878
N 1181 1181 1181 1181
Notes: * indicates significant at 10% level, ** indicates significant at 5% level and

21
Results

• Vendors are 56% faster when T = 0

compared to when T > 0.
• On an average the disclosure happens
in our sample in 20 days.
• If we believe that the effect is linear then
on an average, one day decrease in the
disclosure window increases the
patching speed by 2.8%.

22
Conclusions

• We find that disclosure has significant and

expected result on vendor’s patching
behavior.
• There is a significant CERT effect.
Involvement of CERT leads to faster patching
time irrespective of disclosure.
• Open source vendors patch faster; more
severe vulnerabilities are patched faster and
there is a significant post 9/11 effect.
23