Professional Documents
Culture Documents
Modules Program
19 August 2009
Richard Volkert
US Navy, SSC - Pacific
Eric Forbes
Northrop Grumman
Ken Michaud
US Navy, PEO LMW / PMS 420
Peter Gentile
Northrop Grumman
Tom Sondi
Northrop Grumman
Overview
The System of Systems Challenge SRL Methodology Review Application to Development Status Monitoring Application to Technology Insertion Examining Connections to Cost and Performance
effective SoSE methodology should prepare decision-makers to design informed architectural solutions for SoSE problems.
-Wikipedia
3
The result of this acquisition management challenge has been significant schedule and cost overruns in SoS programs
4
Surface Warfare
5
Mine Countermeasures
(MCM)
Anti-Submarine Warfare
(ASW)
(SUW)
{
Vehicles
Mission Systems
Support Equipment
Mission Modules
}+
Mission Package
Weapons
AMNS AMNS NLOS ALMDS Support Containers Support Equipment Standard Interfaces MPCE Software Crew Detachments
- Mission Modules - Aviation
H-60
Sensors
AQS-20A
COBRA
UDS
TRL Shortcomings
Application of TRL to systems of technologies is not sufficient to give a holistic picture of complex system of systems readiness
TRL is only a measure of an individual technology
Assessments of several technologies rapidly becomes very complex without a systematic method of comparison Multiple TRLs do not provide insight into integrations between technologies nor the maturity of the resulting system
Yet most complex systems fail at the integration points
Individual Technology
System of Technologies
Yes
NO
APPROACH
System System Readiness Readiness Levels Levels (SRL) (SRL) Overall system maturity appraisal
10
Technology 44 Technology
Technology 55 Technology
Technology 44 Technology
Technology 55 Technology
Technology 66 Technology
Technology 66 Technology
Technology 77 Technology
Technology8 Technology8
Technology9 Technology9
Technology 77 Technology
Technology8 Technology8
Technology9 Technology9
Emphasis is on the proper depiction of hardware and software integration between the components Initial Architecture Definition and Setup
String analysis allows for the option of weighting the most important components and evaluation of alternate operational states
Step 4: Apply detailed TRL and IRL evaluation criteria to components and integrations
Checklist style evaluation allows for the ability to take-credit for steps that have taken place beyond the current readiness level
Input TRL and IRL evaluations into algorithm to compute an assessment of overall system status via SRLs
Populate reporting chart templates with evaluation and calculation outcomes to highlight both current status and performance over time
11
SRL Calculation
The SRL is not user defined, but is instead based on the outcomes of the documented TRL and IRL evaluations Through mathematically combining these two separate readiness levels, a better picture of overall complex system readiness is obtained by examining all technologies in concert with all of their required integrations
SRL = IRL x TRL
IRL11 SRL1 SRL2 SRL3 = IRL12 IRL13 IRL12 IRL22 IRL23 IRL13 IRL23 IRL33 x TRL1 TRL2 TRL3
These values serve as a decision-making tool as they provide a prioritization guide of the systems technologies and integrations and point out deficiencies in the maturation process
12
TRL1 = 9
1 0
TRL2 = 6
IRL2,3 = 7 SRL1
Component SRL =
0.54
0.43
0.59
LEGEND
MP Technology Sea Frame System Current Mission Package SRL Status Previous Mission Package SRL Status 1 1 1 Current Mission System SRL Status Technology Readiness Level Integration Maturity Level System Readiness Level Demarcation Scheduled Position Risk to Cost and/or Schedule Low
Tech 2 Tech 1 Tech 3
6 Technology Technology 2 2 7
6 Technology Technology 3 3
Medium
High
SRL
.1
.2
.3
.4
.5
.6
.7
.8
.9
Concept Definition
Feasibility Demonstration
Technology Testing
System Integration
Qualification Testing
DT / OT Complete
14
For complex systems, the amount of information obtained from the SRL evaluation can be overwhelming To maximize applicability SRL outputs are tied to key, program- specific development milestones Progress against these milestones provide key insight to the user regarding current program status, risk and progress
Decision making
Allows components identified as lagging to be analyzed further for root cause Resources can be more properly distributed to those technologies in need Impacts can be examined by quickly analyzing multiple what-if scenarios Allows projected maturity changes to be examined along with cost and schedule
In complex SoS efforts it is not always immediately clear where resources should be applied for maximum gains in maturity and reductions in risk
16
UTAS UTAS //6 MSOBS MSOBS Cntrl Cntrl & & Proc Proc
MP SRL 0.39
MP SRL
0.35
LEGEND
MP Technology Sea Frame System Current Mission Package SRL Status Previous Mission Package SRL Status Current Mission System SRL Status Technology Readiness Level Integration Maturity Level System Readiness Level Demarcation Scheduled Position
USV Cntrl
A component has fallen behind its planned development progression, creating a programmatic risk
.5 .6 .7 .8 .9 1
SRL 17
.1
.2
.3
.4
MP SRL
0.46 0.39
0.45 0.35
7
5
6 MPS MPS 5
LEGEND
MP Technology Sea Frame System Current Mission Package SRL Status Previous Mission Package SRL Status
5
5 5
Current Mission System SRL Status Technology Readiness Level Integration Maturity Level System Readiness Level Demarcation Scheduled Position Risk to Cost and/or Schedule Low Medium High 5
5
3
5
3 5
5
3 6 5 Sea Sea Frame Frame MVCS MVCS
6 months later
USV Cntrl; UTAS / MSOBS Mission Cntrl & Proc; MPS; UDS Cntrl & Planning; MVCS; CM/DF; Proc
SRL 18
.1
.2
.3
.4
.5
What is the projected impact on performance? (How do we optimize?) Are there any legacy design constraints that will impact selection?
20
Design Solution:
Leveraged completely mature and well understood component technologies in a new design
Outcome:
Fielded prototype experienced four years of braking performance issues and derailments causing repeated withdrawals from service
21
SOURCE: Fraser, G.R., Leary, R.J., Pellegrini, M.M.C., Integrating New Light Rail Vehicle Technology in Mature Infrastructure, Transportation Research Circular EC-058, 9th National Light Rail Transit Conference.
infrastructure that has deviated from original design specifications over years of use
In all cases the design met requirements, but failed to adequately accommodate the constraints imposed by the overall system and environment
Performance of a technology in a stand-alone environment does not mean that the technology can be inserted at the system level without significant planning, monitoring, and assessment
22
MP SRL
6 6 MPCE MPCE 6
6 US3 US3
0.57
LEGEND
7 AN/AQS-20A AN/AQS-20A MP Technology Sea Frame System Current Mission Package SRL Status 7 6 6 6 7 AN/ASQ-235 AN/ASQ-235 (AMNS) (AMNS) 1 7 1 1 7 7 MH-60 MH-60 MPS MPS 7
Memory Card
Previous Mission Package SRL Status Current Mission System SRL Status Technology Readiness Level Integration Maturity Level System Readiness Level Demarcation Scheduled Position
7 TSCE TSCE
Hard Drive
MH-60S MH-60S
Risk to Cost and/or Schedule Low 7 7 AN/AES-1 AN/AES-1 (ALMDS) (ALMDS) Medium High
MVCS (RMMV)
US3; BPAUV
AMNS; ALMDS
SRL 23
.1
.2
.3
.4
.5
.6
.7
.8
.9
6 6 BPAUV BPAUV
MP 1 MP SRL
w/o Sea Frame
MP SRL
6 6 MPCE MPCE 6
6 6
6 MVCS MVCS (USV) (USV) 6 USV USV 7 DLS DLS (RMMV) (RMMV) 7 AN/WLD-1 AN/WLD-1 (RMMV) (RMMV) 7 6
6 US3 US3
0.64 0.60
0.67 0.57
LEGEND
7 DLS DLS (On-board) (On-board) 5 7 AN/AQS-20A AN/AQS-20A MP Technology Sea Frame System Current Mission Package SRL Status 7 7 AN/ASQ-235 AN/ASQ-235 (AMNS) (AMNS) 1 1 1 9 9 MH-60 MH-60 MPS MPS 9
Memory Card
Previous Mission Package SRL Status Current Mission System SRL Status Technology Readiness Level Integration Maturity Level System Readiness Level Demarcation Scheduled Position
MH-60S MH-60S
Risk to Cost and/or Schedule Low 7 7 AN/AES-1 AN/AES-1 (ALMDS) (ALMDS) Medium High
DLS (OB)
USV BPAUV BPAUV PC US3
DLS(RMMV)
MPCE
RMMV
AQS-20 MH-60S
SRL 24
.1
.2
.3
.4
.5
.6
.7
.8
.9
Cost Profiles
PEO LMW / PMS 420 is working with NAVSEA 05C (NAVSEAs cost analysis division) to develop a life cycle cost model specifically tailored to SoS analysis Factors contributing to costs in SoS
Integration type (physical, functional, logical) Use of standards (Were components designed to integrate?) Maturity of technologies being integrated
A correlation between the SRL and cost numbers brings about the ability to track actual development maturity vs. costs Linkage to technology trade-off and planning environments allows cost to be analyzed in consideration with maturity and performance Concept being explored by Stevens Institute would allow for overall program cost estimate funding allocations to be discredited over the development cycle
26
SRL
Functional
Physical
Logical
.1 Jun 2006
.2 Jan 2007
.3 Jul 2007
.4 Nov 2007
.5 May 2008
.6
.7 Aug 2009
.8 Apr 2010
.9 Jan 2011
1 Apr 2012
Materiel MILESTONE A Development Alternative Decision Systems Initial Review Technical Review
MILESTONE C System Verification Review Functional Configuration Audit Production Readiness Review Technology Readiness Assessment 2
Initial Operational Capability Physical Configuration Audit Full Rate Production Decision Review
Current Mission Package SRL Status by View (Functional, Physical, Logical) .3 System Readiness Level Demarcation Scheduled Position (IMS)
SRL assessment and test events / milestone gates are at or in advance of schedule SRL assessment is at or in advance of schedule, but test events / milestone gates remain to be closed
27
SRL assessment and test events / milestone gates are behind schedule
KPPs, and TPMs, which can then be traded against cost and maturity
CRL-18
A
Req1 Req2 Req4 Req3
28
Conclusions
SoS development represents a new level of challenge in acquisition management SRL provides one possible assessment, analysis and management technique Methodology leads to holistic monitoring of many factors impacting system development
Capturing integration is key
Future work includes extending the concepts for understanding cost and performance impacts in an incremental acquisition
29
QUESTIONS?
Back-up
Populate these matrices with the appropriate values from the previously documented TRL and IRL component evaluations and then normalize to a (0,1) scale by dividing through by 9 For an integration of a technology to itself (e.g. IRLnn) a value of 9 should be placed in the matrix For an instance of no integration between technologies a value of 0 should be placed in the matrix
[TRL ]n1
[IML ]nn
32
Decision Support Metrics for Developmental Life Cycles, Users Guide: Version 2.0, Northrop Grumman Corp. and Stevens Institute of Technology, 5 September 2007
SRL1 IML11TRL1 + IML12TRL2 + ... + IML1nTRLn SRL IML TRL + IML TRL + ... + IML TRL 22 2 2n n [SRL ] = 2 = 21 1 ... ... IML TRL IML TRL IML TRL + + + SRL ... n n1 1 n2 2 nn n
33
Decision Support Metrics for Developmental Life Cycles, Users Guide: Version 2.0, Northrop Grumman Corp. and Stevens Institute of Technology, 5 September 2007
2 Integrations 1 Integration
34
Decision Support Metrics for Developmental Life Cycles, Users Guide: Version 2.0, Northrop Grumman Corp. and Stevens Institute of Technology, 5 September 2007
SRL
SRL1
SRL2
SRL3
These individual values serve as a decision-making tool as they provide a prioritization guide of the systems technologies and integrations and point out deficiencies in the maturation process The composite SRL for the complete system is the average of all normalized SRL values. (Note that weights can be incorporated here if desired.)
SRLComposite
A standard deviation can also be calculated to indicate the variation in the system maturity
35
Decision Support Metrics for Developmental Life Cycles, Users Guide: Version 2.0, Northrop Grumman Corp. and Stevens Institute of Technology, 5 September 2007
Divide by 9
Remember a technology integrated with itself receives an IRL value of 9 (e.g. IRL11), while technologies for which there is no connection between them receive a value of 0 (e.g. IRL13).
0 .78 1.0
SRL for System Alpha Calculating the SRL and Composite Matrix
SRL = IRL x TRL
Component SRL SRL1 SRL2 SRL3 SRL1 SRL2 SRL3 = = 1.07 0.54 1.30 0.43 1.19 0.59
(0,n) scale
Where n is equal to the number of integrations for that technology
(0,1) scale
Both individual and composite scores provide key insights into the actual maturity of the system as well as where risk may lie and attention directed for greatest benefit
37
Sauser, B., J. Ramirez-Marquez, D. Henry and D. DiMarzio. (2007). A System Maturity Index for the Systems Engineering Life Cycle. International Journal of Industrial and Systems Engineering. 3(6). (forthcoming)
Operational strings were created that identified the components required to utilize a single function of the system Assessment of the SRL for each of these options allows for a better understanding of the maturity of each operating configuration Understanding the true status of the system on an operational string level allows for the opportunity to field initial capability earlier and then add to it as other strings mature
38
39
40
What is an IRL?
A systematic measurement reflecting the status of an integration connecting two particular technologies
IRL Pragmatic Definition
Integration is Mission Proven through successful mission operations. Actual integration completed and Mission Qualified through test and demonstration, in the system environment. The integration of technologies has been Verified and Validated with sufficient detail to be actionable. The integrating technologies can Accept, Translate, and Structure Information for its intended application. There is sufficient Control between technologies necessary to establish, manage, and terminate the integration. There is sufficient detail in the Quality and Assurance of the integration between technologies. There is Compatibility (i.e. common language) between technologies to orderly and efficiently integrate and interact. There is some level of specificity to characterize the Interaction (i.e. ability to influence) between technologies through their interface. An Interface between technologies has been identified with sufficient detail to allow characterization of the relationship.
9 8 7 6 5 4
Syntactic
Semantic
3 2 1
41
Gove, R. (2007) Development of an Integration Ontology for Systems Operational Effectiveness. M.S. Thesis. Stevens Institute of Technology. Hoboken, NJ
9 IML
0.00 5 1 5 9 TRL
9 IML
0.00 5 1 5 9 TRL
42
TRL 1 3 5 7 9
TRL 1 3 5 7 9
TRL 1 3 5 7
43
TRL 1 3 5 7 9
8.6
7.9
7.9
7.2
9.0
8.7
8.6
8.0
8.6
7.9
8.8
8.5
9.0
8.7
8.6
8.0
MH-60S - MPCE
8.6
8.4
7.7
8.1
Used by 5 Threads
9.0
8.8
8.6
8.4
COBRA - VTUAV
8.6
8.9
8.8
8.9
Used by 1 Thread
9.0
9.0
8.6
8.9
Comparative Sensitivity A look at how the algorithms penalized the SRL rating relative to one another (1 is most severe)
Non-connected, Self IRLs = 0
Sys String
1,2 3 1,2 4
2 1 3,4 3,4
1 2 3 4
1,2 3 1,2 4
1,2 3 1,2 4