Quantitative Risk Management

Tom Tuduc


1 09/25/08

         

Risk is ubiquitous - We are all familiar with risks: RISK board games and video games. Download risk-free product trials Buy products that reduce risks of illness Terrorist threats Take calculated risks (video clip ReturnOfTheKing) Avoid running the risk of ... Risk attitude Eliminating risk by getting more information and/or controlling outcome (video clip ValueInfoControl) Who is at risk, high risk groups (video clip highrisk)


2 09/25/08


Homeland Security is complex and include uncommon and/or hypothetical uncertainties. It takes both qualitatively and quantitatively models to consider hundreds of intelligences with different credibility and accuracy. Topics: 1. How Influence Diagrams/Decision Analysis help experts communicate and model Homeland Security decisions, threats, and countermeasures visually, qualitatively, and quantitatively. 2. How Decision Analysis enable calculations that maximize security, make decision policies, quantify insights of each threat factor, and the worth of additional information and control on each factor. 3. A review of several examples in the literature: influence diagrams in terrorist threat countermeasures, early warming systems, toxin containment policies, and intrusion-aware information systems. 4. Security categories, Application trends, technology integration possibilities, and online resources


3 09/25/08

Table of Content
                       

Introduction Summary Table of Content What is Security Analytics? Security Risk Methodology - the Four Steps Risk Management: Dealing with Uncertainty Example of Security Application Areas Characteristics of complex risk problem, Decision Analysis & Influence Diagrams Tradeoffs & Risk Preference Differences between Trees and Diagrams Certainty Equivalence, Utility & Risk Premium Risk taking Risk averse Tutorial Example Best Policy and value of Control Risk Profiles Gaining Insights Sensitivity Analysis Similar security ROI starting point Similar Intrusion Detection problem A more complex party problem A more complex security ROI problem Complex Intrusion-Aware Model

             

Homeland Security Infrastructures & Assets Homeland Security - System View Homeland Security - Decision View Infrastructure Elements Homeland Security Decision Analysis & Influence Diagrams Exam Example 1 Overarching Influence Diagram Example 2: Site Profiler Architecture & Influence Network Example 3 – Using Analytica Security Categories Where are the numbers Tools & Resources Conclusion
4 09/25/08

 

DEFINITION: Security Analytics (Table1) are the use of analytics to optimize security and security ROI. Applications: – Model – Processes – Policies – Systems
Probability Statistics

Stochasti c Markov

Dynamic programming Game theory

Graph theory Information theory


Utilit y theor y

5 09/25/08

Security Methodology - the Four Steps

1. Determine risk:
– Assets and risk to assets. – Making security ROI known.

 

2. Analyze risk: * – Qualitative – Quantitative: Analytics 3. Design and Implement: policies, architectures, technologies, trainings,
and countermeasures

4. Management: Monitoring, audits, and evaluate

* "One of the major problems is that security risk assessment and the benefits of using the

results of risk assessment cannot be measured in any sufficiently accurate to provable way... Positive benefit is absence of unknown possible loss" Tom Peltier, "Risk Analysis Vs. Security Controls." NetSec 2002


6 09/25/08

Risk Management: Dealing with Uncertainty

Fundamental Approaches  Frequentist – Based on hundreds or thousands of events. – Probability lies objectively in the world, not in the observer.

Bayesian – Based on personal experience. – Probability is different for people having different past experiences.


7 09/25/08

Example of Security Applications

     

Security ROI Risk assessment and management Knowledge management and Information retrieval (1) SPAM filtering (2) Intrusion Detection Systems Other examples: Search engines, portfolio management, polling, etc.

(1) 21 US agencies with 200,000 employees has deployed Autonomy, a knowledge management tool based on Bayes and Shannon theorems, for homeland security functions (Business Weekly, 31 October 2002.) (2) Baysian-based SPAM filters: http://www.webarches.com/filters.html


8 09/25/08

Characteristics of complex problems

 

Many uncertainty/probabilities cannot be obtained from empirical frequency distributions because the events are uncommon and/or hypothetical. Probabilities come from expert opinions with different experience of the same problem In a closed-loop system, the probabilities improves over time with repeated cycles. Time is a luxury not always available.


9 09/25/08

Decision Analysis/ Influence Diagram (DAID)



risks, decisions, threats, and countermeasures


visually, qualitatively, and quantitatively.


10 09/25/08

    

Decisions: made by the decision maker Uncertain events: events with discrete outcomes or probability functions Consequences: values resulting from the decisions and uncertain events outcomes Risk Preferences: how the decision maker feels about the consequences (1) Objectives: direction and value, i.e. eliminate risk areas, maximizing ROI, minimizing loss of data and/or resources.

(1) Will the real risk-preference stand up: A popular misconception is that security managers in private sectors are risk-averse and overspend on security. IDC research data shows otherwise. A typical organization of 5,000 employees, on average spends $1 million on security products ($200/person, or $500 for each $1 million in revenues).


11 09/25/08

Tradeoffs & Risk Preference

 

Conflicting objectives: A policy may be optimal for one objective, but not for all objectives, i.e. how much expected loss of data availability is an agency willing to accept to increase data integrity to 100 percent. Tradeoffs (conflicting objectives): 10 percent loss in data integrity is equivalent to 50 percent loss in data availability Risk Preference: which Risk Profile is your organization's

•High risk-taker: black •Low risk-taker :blue

(video clip riskProfiles)
12 09/25/08


Differences between Decision Trees and Influence Diagrams

Influence diagrams show dependencies among variables clearly: good visuals for communication and qualitative relationship. Influence diagrams are compact one or two order of reduction in node representation in typical problems. Decision trees show details of possible paths/scenarios: relatively good visual for small problems. Best for quantitative calculations. Decision trees show asymmetric outcome trees.


13 09/25/08


14 09/25/08

Certainty Equivalence, Utility & Risk Premium

    

Common decision rule: maximize expected value, often expected monetary value. However, this is not realistic for the risk-averse. Better decision rule: Expected Value with minimum risk variance (portfolio investment) Best decision rule: maximize expected utility. Utility is found by presenting simple lotteries to decision makers. Certainty Equivalence: taking monetary equivalence instead of playing the lottery. Risk premium: EV of lottery - CE of lottery


15 09/25/08


  

Risk premium = EV - CE or -$2. CE is larger than EV Buying a lot of superlotto tickets is risk taking
16 09/25/08



  

Even though EV is higher now (50 versus 23), Certainty equivalence is lower (25 versus -5) Risk premium = 50 - 5 = $55. CE is less than EV

This is analogous to hiring a consultant, or outsourcing instead of performing a function internally.
17 09/25/08


Tutorial Example


18 09/25/08


19 09/25/08

Best Policy and Value of Control

The Influence Diagram and Decision Tree show the Location Decision is made independent of knowing the weather

 

If we don't know what the weather will be, we should locate it on the porch because that has the highest payoff of $38 million. Best case saving: (60-38) or $22 million. This is Value of Control *

* The Department of Energy benefits by eliminating security-update risks (Value of Control) when it required Oracle to deliver its 9i database with all security features and to take responsibility of maintaining security updates. This is an unusual but excellent example of cyber-security practice.


20 09/25/08

Risk Profiles

Locating the party by the pool can give negative utility if it rains. But if it’s sunny it’s the best decision. If it’s cloudy, it might rain.


21 09/25/08

Gaining Insights

1. If we know what the weather will be, we can make a better decision. Thus the new expected payoff is now $47 million, instead of $38 million.

2. If we want to ask a security expert (clairvoyant) about what the weather will be, we should only pay a maximum of $9 million (new expected payoff - old expected payoff) – New expected payoff: (.2*45) + (.5*40) + (.3*60)= 47 – Old expected payoff: $38

3. New Value of Control: new best case saving is (60- Expected Value) = 60-47 = 13
22 09/25/08


Sensitivity Analysis


23 09/25/08

Similar Security ROI starting point

Budgets: basic security (firewall, VPN, antiVirus), audits, realtime intrusion detection, advanced access control, encryption, etc.
24 09/25/08


Similar Intrusion Detection problem


25 09/25/08


26 09/25/08


27 09/25/08

Complex Intrusion-Aware Model
TRIAD (Trustworthy Refinement through Intrusion-Aware Design): an intrusionaware model developed at CMU/SEI (TECHNICAL REPORT CMU/SEI-2003TR-002) SUMMARY PROBLEM: Military and business systems face increasingly sophisticated and coordinated computer network attacks. Existing security system development are typically isolated solutions resulting in patchwork designs that are not robust under attack. TRIAD, a model, helps IT decision makers to formulate and maintain a coherent and justifiable survivability strategy that addresses mission-compromising threats. TRIAD uses DAID to model the dynamics of fraud and authentication. TRIAD's goals are:
    

Develop a development methodology for security systems to resist, recognize, recover from, and adapt to mission-compromising attacks. to provide a documented response to the primary threats to the mission; to provide a justification for and the limitations of the system design; to support the design and implementation of the desired system behavior across multiple systems and multiple development teams; and to support maintenance and evolution as the system operations and threat environment evolve over time.
28 09/25/08



29 09/25/08

Homeland Security Infrastructures & Assets

Critical Infrastructures

Key Assets

– – – – – – – – – – – – – –

Agriculture Food Water Public Health Emergency Services Government Defense Industrial Base Information and Telecommunications Energy Transportation Banking and Finance Chemical Industry and Hazardous Materials Postal and Shipping

– – – – –

National Monuments Dams Nuclear Power Plants Government Facilities Commercial Key Assets


30 09/25/08

Homeland Security- System View


31 09/25/08

Homeland Security- Decision View


32 09/25/08

Infrastructure elements


33 09/25/08

Homeland Security Decision Analysis & Influence Diagrams Examples

2. 3.

4. 5. 6.

Probabilistic Modeling of Terrorist Threats: A Systems Analysis Approach to Setting Priorities Among Countermeasures Site Profiler, a system being used in bio-terrorism early warning systems, passenger and cargo profiling, vulnerability assessments, threat warnings and dissemination. Using Analytica: Toxin Containment Model and Analysis TRIAD (Trustworthy Refinement through Intrusion-Aware Design): an intrusion-aware model Others: GIS and Decision Analysis Journal, COPLINKS (Search and match given incomplete information), Paper "Warning and Response in Homeland Security“, and Sandia/CA’s Weapons of Mass Destruction Decision Analysis Center


34 09/25/08

Example 1 & Influence Diagram
Probabilistic Modeling of Terrorist Threats: A Systems Analysis Approach to Setting Priorities Among Countermeasures. Elisabeth Paté-Cornell and Seth Guikema. Department of Management Science and Engineering. Stanford University. Military Operations Research, Vol. 7, No 4, pp. 5-20 December 2002. SUMMARY PROBLEM: assess the benefits of risk reduction by different countermeasures and their costs OBJECTIVE: - Prioritize the protection of US infrastructures, networks and socio-economic components - Discover most effective means of reducing the overall threat, i.e. the disruption of the terrorists’ supply chain - Prioritize intelligence information that needs to be gathered given accuracy, time, and constraints.


35 09/25/08


36 09/25/08

Example 2 – Architecture and Diagram
Site Profiler, a system developed by Bryan Ware, Anthony Beverina, Lester Gong, and Brian Colder at Booz Allen Hamilton and Digital Sandbox. Site Profiler is used in bio-terrorism early warning systems, passenger and cargo profiling, vulnerability assessments, threat warnings and dissemination. Site Profiler applies DAID to combined data from various sources. SUMMARY PROBLEM: Build a system to sift through massive amount of information to determine terrorist risk OBJECTIVE: Determine the following:  how likely a terrorist will attempt attacks including tactic, weapon, delivery system against an asset  how likely the terrorist will succeed  consequences of successful attacks CHALLENGES:  High volumes of data.  Disparate sources of data and information  Diverse forms of information  Significant organizational friction among producers, owners, and consumers of information
37 09/25/08



38 09/25/08


39 09/25/08

Example 3 – Using Analytica
Using Analytica to model and analyze the cost and benefit of Toxin Containment (Adopted from Analytica’s Toxic Emission Control) SUMMARY PROBLEM: Determine costs and benefits of containing an airborn toxin that is potentially fatal. Objective: Maximize the expected benefit, defined as benefits(1) less the cost(2) to contain toxin. (1) Cost to contain toxins depends on the containment level (logarithmic) (2) Benefits as the reduced mortality multiplied by the value of a life


40 09/25/08

Problem: how much to contain and eliminate certain toxins including the option of reducing them by zero.


41 09/25/08


42 09/25/08

Security Categories
             

Access Controls, Authentication Anti-eavesdropping Anti Virus Virus protection/detection Automated Patch Management Biometrics Authentication of users/terminals Business Continuity & Disaster Recovery Content Delivery Network Security Email spam filters Encryption Extranet Security Integration Firewalls and Internet Security Intrusion Detection & Network Monitoring Media Security Destruction Devices

           

Media Protection Safes Media Security Physical/Facility Security - AntiTheft Devices Physical/Facility Security -Entrance Control Systems Physical/Facility Security Environmental Controls Physical/Facility Security - Power Management Risk Management Risk Analysis Security Incident Management Single Sign On Software Controls Telecom & Remote Access Security Wireless Security
43 09/25/08


Where to find statistics

 

1. Symantec Internet Security Threat Report Volume IV - Every six months. – During the first half of 2003, Symantec saw a 50% increase in confidential data attacks using backdoors. – In the past six months, Web application vulnerabilities increased 12 percent, malicious codes were up 20 percent, and worms and viruses increased 19 percent 2. Computer Security Institute/ FBI Computer Crime and Security Survey - Yearly 3. @Stake Advisories and Research Labs (see Table below)


44 09/25/08


Traditional Decision Analysis and/or Influence Diagrams: Analytica, DATA, Decide, DecisionPro, DPL, Expression Tree, Precision Tree, Risk Detective, Supertree/Sensitivity, TreePlan.

Risk management tools:
Analytica, DLP,LHS, Fuldek,SAPHIRE, SETS, SANET, SABLE, FTAP, SEATree, Stepwise,


45 09/25/08

The End
"[T]he U.S. Air Force …is faced with a multitude of decisionsprogrammatic, technical, personnel, strategic, and yes, cultural - that we must make based on knowledge of, and respect for, the relevant underlying data. In that spirit … operations research and decision analysis are and will continue to contribute to national security decisionmaking."- Secretary of the Air Force James G. Roche, OR/MS December 2002


46 09/25/08

Sign up to vote on this title
UsefulNot useful