The Software Assurance

Metrics and Tool Evaluation

(SAMATE) Project

Paul E. Black
Computer Scientist, NIST
paul.black@nist.gov
+1 301-975-4794
OWAS
P
AppSe
c
This is a work of the U.S. Government and is not subject to
copyright protection in the United States.

DC
October 2005
The OWASP
http://www.owasp.org/
Foundation
Outline

Overview of the NIST SAMATE project

Purposes of tool and technique evaluation
Software and effectiveness metrics
Report of workshop on Defining the State
of the Art in Software Security Tools
Final comments

OWASP AppSec DC 2005 2

The NIST SAMATE Project

 Surveys
 Tools
 Researchers and companies
 Workshops & conference sessions
 Taxonomy of software assurance (SwA) functions &
techniques
 Order of importance (cost/benefit, criticalities, …)
 Gaps and research agendas
 Studies to develop metrics
 Enable tool evaluations
 Write detailed specification
 Develop test plans and reference material
 Collect tool evaluations, case studies, and comparisons
http://samate.nist.gov/
OWASP AppSec DC 2005 3
Taxonomy of SwA Tool Functions and
Techniques

 Concept or business need

 Use cases
 Changes to current systems
 Requirements and design
 Consistency
 Completeness
 Compliance
 Implementation
 The usual …
 Assessment and acceptance
 External
 Automated vulnerability scanners
 Penetration test assistants
 Other standard testing techniques: usage, spec-based, statistical, worst-case/criticality, etc.
 Insider
 Automated code scanners
– Syntactic, e.g., “grep”
– Semantic
 Code review assistants
– Source code
– Virtual Machine code (e.g., Java bytecode or .Net intermediate code)
– Binary (debugger, decompiler)
 Operation
 Operator training
 Auditing
 Penetration test

OWASP AppSec DC 2005 4

Planned Workshops

 Enumerate SwA functions and techniques

 Approach (code vs. spec, static vs. dynamic)
 Software type (distributed, real time, secure)
 Type of fault detected
 Recruit focus groups
 Which are the most “important”?
 Highest cost/benefit ratio?
 Finds highest priority vulnerabilities?
 Most widely used?
 Critique reference dataset
 Identify gaps in functions and recommend research
 Plan and initiate studies for metrics

OWASP AppSec DC 2005 5

Outline

Overview of the NIST SAMATE project

Purposes of tool and technique evaluation
Software and effectiveness metrics
Report of workshop on Defining the State
of the Art in Software Security Tools
Final comments

OWASP AppSec DC 2005 6

Purposes of Tool or Technique
Evaluations
Precisely document what a tool does (and
doesn’t) do
… in order to …
Provide feedback to tool developers
Simple changes to make
Directions for future releases
Inform users
Match the tool or technique to a particular
situation
Understand significance of tool results
Know how techniques work together
OWASP AppSec DC 2005 7
Developing a Specification

 After a technique or tool function is selected by

the working group …
 NIST and focus group develops a clear, testable
specification
 Specification posted for public comment
 Comments incorporated
 Develop a measurement methodology:
 Test cases
 Procedures
 Reference implementations and data
 Scripts and auxiliary programs
 Interpretation criteria

OWASP AppSec DC 2005 8

 SAMATE Project Timeline
1 2 3 4 5 6 9 12 15 18 21 24
Function Tool Survey
Taxonomy Survey Publication

tool
Workshop1 testing Workshop 2 Workshop 3
SwA matrix research metrics
classes gaps studies

focus focus focus focus

group group group group
class 1 class 2 class 1 class 2
Spec0
select func
test plan
Spec1
strawman
spec test plan
draft

select func Spec0

strawman Spec1
spec

draft
Why Look at Checking First?

Vital for software developed outside, i.e.,

when process is not visible
Applicable to legacy software
Feedback for process improvement

Process experiments are expensive

Many are working on process (SEI, PSP,
etc.)

OWASP AppSec DC 2005 10

Outline

Overview of the NIST SAMATE project

Purposes of tool and technique evaluation
Software and effectiveness metrics
Report of workshop on Defining the State
of the Art in Software Security Tools
Final comments

OWASP AppSec DC 2005 11

But, is the tool or methodology
effective?
 Is this program secure (enough)?
 How secure does tool X make a program?
 How much more secure does technique X make a
program after techniques Y and Z ?
 Do they really find or prevent bugs and
vulnerabilities?
 Dollar for dollar, does methodology P or S give
more reliability?

OWASP AppSec DC 2005 12

Toward Software Metrics

Qualitative comparison
warmer, colder buggy, secure

Formally defined quantity

temperature quality? confidence?

Unit and scale

degree, Kelvin ?

Measured value
Derived units
Heat energy=smt Software assurance
≈ pt
OWASP AppSec DC 2005 13
Benefits of SAMATE Project

 Define metrics for evaluating SwA tools

 Users can make more informed tool
choices
 Neutral test program
 Tool creators make better tools

OWASP AppSec DC 2005 14

Outline

Overview of the NIST SAMATE project

Purposes of tool and technique evaluation
Software and effectiveness metrics
Report of workshop on Defining the State
of the Art in Software Security Tools
Final comments

OWASP AppSec DC 2005 15

Workshop on Defining the State of the
Art in Software Security Tools
 Workshop Characteristics
 NIST, Gaithersburg
 10 & 11 August 2005
 http://samate.nist.gov/softSecToolsSOA
 45 people from government, universities, vendors and service providers,
and research companies came
 Proceedings, including discussion notes and submitted material, should
be available from above URL when you see this.
 Goals
 Understand the state of the art of software security assurance tools in
detecting security flaws and vulnerabilities.
 Discuss metrics to evaluate the effectiveness of such tools.
 Collect flawed and “clean” software for a reference dataset.
 Publish a report on classes of software security vulnerabilities.

OWASP AppSec DC 2005 16

Outcomes of Workshop I

Understand the state of the art of software

security assurance tools in detecting
security flaws and vulnerabilities.
A report is being written.
Discuss metrics to evaluate the tool
effectiveness
All agreed that software metrics and tool
effectiveness metrics are a good idea.
No consensus on how to approach the
challenge.

OWASP AppSec DC 2005 17

Outcomes of Workshop II

 Collect flawed and “clean” software to be a

reference.
 Several collections emerged: MIT, Fortify, etc.
 Attendees agreed that a shared reference dataset would
help.
 NIST reference dataset in development.
 Prototype available at (URL forthcoming)
 Report on classes of software security
vulnerabilities
 Discussed several existing flaw taxonomies: Clasp,
PLOVER (CVE), etc.
 Attendees agreed a common taxonomy would help.
 Discussions continuing on samate email list.

OWASP AppSec DC 2005 18

Outline

Overview of the NIST SAMATE project

Purposes of tool and technique evaluation
Software and effectiveness metrics
Report of workshop on Defining the State
of the Art in Software Security Tools
Final comments

OWASP AppSec DC 2005 19

Society has 3 options:

Learn how to make software that

works

Limit size or authority of software

Accept failing software

OWASP AppSec DC 2005 20

Contact to Participate

 Paul E. Black
Project Leader
Software Diagnostics & Conformance Testing
Division, Software Quality Group, Information
Technology Laboratory, NIST
paul.black@nist.gov

OWASP AppSec DC 2005 21