ATLAS Shrugged

Pat O’Toole
(with apologies to Ayn Rand and John Galt)

ATLAS
“Ask The Lead AppraiserS” A scenario-based email forum used to elicit opinions on “interesting” topics Distributed to all 400+ LAs and 1500+ non-LAs Limited to one page Multiple choice format - ample room for comments Results are compiled and published by PACT with no intellectual property rights retained SEI is just another recipient of the data – they do not sponsor or influence ATLAS in any way.
2

ATLAS Scenarios
Scenario number (# LA’s / # non-LA’s) #1 – Bidirectional traceability (46/45) #2 – Process descriptions/measures (40/17) #3 – Applicability of SAM (66/75) #4 – Interpretational issues (79)

3

ATLAS Scenarios #1 – Bidirectional traceability (46/45) Manage Requirements #2 – Process descriptions/measures (40/17) #3 – Applicability ofObtain (66/75) SAM Manage Obtain an Commitment Requirements Understanding to #4 – Interpretational issues (79)Changes of Requirements Requirements Maintain Bidirectional Traceability of Requirements Requirements Identify Inconsistencies Between Project Work and Requirements Traceability Matrix 4 .

000 person hours 14 months into its 18 month schedule preparing to initiate system testing No alternative practices for this project.ATLAS #1 – Bidirectional Traceability A project in a SCAMPI A appraisal is: estimated to be 30. 5 .

4? A. For this project. B. E. Vertical and horizontal traceability Either vertical or horizontal traceability Only vertical traceability Only horizontal traceability Other 6 .Question 1 1. C. which selection best represents your view of model expectations with respect to REQM SP1. D.

Answer 1: Model Expectations Selected choice: A: Vertical and horizontal B: Vertical or horizontal C: Vertical only D: Horizontal only E: Other/None Leads Non-Leads 70% 84% 13% 0% 13% 11% 0% 4% 4% 0% 7 .

NI) is most appropriate if there is: A.Question 2 2. PI. ___ Ample evidence of vertical traceability but no evidence of horizontal traceability? B. ___ Ample evidence of horizontal traceability but no evidence of vertical traceability? 8 . What characterization (FI. LI.

PI D. LI C.Characterizations Ample evidence of vertical traceability but no evidence of horizontal traceability Selected choice: A. NI Leads 20% 39% 36% 5% Non-Leads 15% 32% 47% 6% 9 .Answer 2a . FI B.

Answer 2b . PI D.Characterizations Ample evidence of horizontal traceability but no evidence of vertical traceability Selected choice: A. LI C. FI B. NI Leads 7% 11% 66% 16% Non-Leads 2% 9% 66% 23% 10 .

Question 3 3. I don’t have a clue! (Don’t know) 11 . Both E. please indicate if you consider it to be: A. For each of the following. Neither D. Horizontal Traceability C. Vertical Traceability B.

Answer 3a – Traceability Type High-level business requirements are traceable to feature requirements Selected choice: A: Vertical B: Horizontal C: Neither D: Both E. Don't Know Leads 95% 5% 0% 0% 0% Non-Leads 86% 2% 0% 12% 0% 12 .

Don't Know Leads 4% 82% 2% 9% 2% Non-Leads 9% 74% 2% 16% 0% 13 .Answer 3b – Traceability Type Traceability is maintained among interdependent functional requirements Selected choice: A: Vertical B: Horizontal C: Neither D: Both E.

” And we wonder why there are interpretational issues! . Don't Know Leads 78% 16% 2% 4% 0% Non-Leads 67% 21% 4% 9% 0% 14 Note: Traceability ACROSS the life cycle is “vertical.Answer 3c – Traceability Type Each of the 500+ system test cases lists the specific requirement(s) being tested Selected choice: A: Vertical B: Horizontal C: Neither D: Both E.

Answer 3d – Traceability Type System requirements are traceable to the group(s) to which they are allocated Selected choice: A: Vertical B: Horizontal C: Neither D: Both E. Don't Know Leads 51% 29% 16% 0% 4% Non-Leads 32% 39% 14% 16% 0% 15 .

Don't Know Leads 51% 29% 16% 4% 0% Non-Leads 40% 35% 16% 7% 2% 16 .Answer 3e – Traceability Type Technical requirements are traceable to specific elements in the WBS Selected choice: A: Vertical B: Horizontal C: Neither D: Both E.

edu/cmmi/faq/new-faq.sei.cmu.ATLAS #1 Note The SEI’s website contains answers to “Frequently Asked Questions” (FAQ) Bidirectional traceability is covered: http://www.html#Q318 See partial text on next slide Only 1 of nearly 100 respondents (a lead appraiser) mentioned the SEI FAQ! 17 .

When the requirements are managed well.g. horizontal traceability would follow related requirements across two work groups working on two associated components of a product. traceability can be established from the source requirement to its lower level requirements and from the lower level requirements back to their source.. 18 . Horizontal traceability is also important. but it is not required to satisfy bidirectional traceability.SEI FAQ regarding Traceability Vertical traceability identifies the origin of items (e. For example. Horizontal traceability identifies the relationships among related items across work groups or product components for the purpose of avoiding potential conflicts. customer needs) and follows these same items as they travel through the hierarchy of the WBS to the project teams and eventually to the customer.

ATLAS Scenarios #1 – Bidirectional traceability (46/45) #2 – Process descriptions/measures (40/17) #3 – Applicability of SAM (66/75) #4 – Interpretational issues (79) 19 .

ATLAS #2: Process Descriptions When conducting a ML2 appraisal. 20 . and organizational personnel corroborate this. the organization has no documented process descriptions.

Question 1 Would you document a weakness regarding the lack of process descriptions? _____ Yes No Lead Appraisers 79% 21% Non-Leads 87% 13% Do you perceive this to be a goal-threatening weakness? _____ Yes 65% 73% No 35% 27% 21 Lead Appraisers Non-Leads .

PP. 22 .ATLAS #2: Metric Specifications When conducting a ML2 appraisal the organization employs project and product measures. They DO plan and track the process activities associated with REQM. Organizational personnel corroborate this.8 example boxes. but they have not implemented any process measures as suggested by the GP2. etc.. PMC. but no process measures.

Question 2 Would you document a weakness regarding the lack of process measures? _____ Yes No Lead Appraisers 51% 49% Non-Leads 80% 20% Do you perceive this to be a goal-threatening weakness? _____ Yes 18% 53% No 82% 47% 23 Lead Appraisers Non-Leads .

The unspecified measures were already wellestablished and used consistently. In addition to the 4 specified measures. the org has specified only 4 measures: SLOC. Peer Review Defects. 24 . the org and projects capture and use many more measures. Earned Value. Org personnel contend that the specified measures are those that were most recently introduced (throughout the past year). but no specs exist for these additional measures.ATLAS #2 – How Much Is Enough? When conducting a ML2 appraisal. and Test Defects. The specifications are complete and cover all of the MA SG1 specific practices.

Question 3 Would you document a weakness regarding the limited number of specified measures? _____ Yes No Lead Appraisers 70% 30% Non-Leads 53% 47% Do you perceive this to be a goal-threatening weakness? ____ Yes No Lead Appraisers 27% 73% Non-Leads 33% 67% 25 .

ATLAS Scenarios #1 – Bidirectional traceability (46/45) #2 – Process descriptions/measures (40/17) #3 – Applicability of SAM (66/75) #4 – Interpretational issues (79) 26 .

free of charge and “at your own risk. Scenario 1 The project team is incorporating an “Open Source” component. 27 . nor any responsibility for its ongoing support/maintenance.ATLAS #3. The source code was posted on the originator’s website with an indication that it can be used without restriction.” The originator also indicated she retains no intellectual property rights with respect to the component.

Question 1 Must SAM be applied with respect to the Open Source component? _____ Yes Lead Appraisers 35% Non-lead Appraisers 19% No 65% 81% 28 .

maintenance fees. 29 . According to the SOW.ATLAS #3. Scenario 2 The customer’s SOW requires that you incorporate an unmodified version of component X which is available solely from Company Y. the customer will negotiate X’s acquisition cost. and license fees with Company Y.

Question 2 Must SAM be applied with respect to Company Y? _____ Lead Appraisers Non-lead Appraisers Yes 28% 34% No 72% 66% Must SAM be applied with respect to the customer? _____ Yes No Lead Appraisers 50% 50% Non-lead Appraisers 64% 36% 30 .

ATLAS #3. 31 . Scenario 3 The solution that your very small company intends to provide to your customer includes a laser jet printer supplied by Very Big Company. It is off-the-shelf and no modifications are required.

Question 3 Must SAM be applied with respect to Very Big Company? _____ Yes Lead Appraisers 52% Non-lead Appraisers 51% No 48% 49% 32 .

ATLAS Scenarios #1 – Bidirectional traceability (46/45) #2 – Process descriptions/measures (40/17) #3 – Applicability of SAM (66/75) #4 – Interpretational issues (79) 33 .

2-1 Maintain Bidirectional Traceability of Requirements Plan for Data Management Establish Measurement Objectives Establish Estimates of Work Product and Task Attributes Perform Configuration Audits 43 21 19 16 16 54% 27% 24% 20% 20% 1 2 3 4 4 18% 27% 35% 41% 48% 34 . % of Number of ML2 within ML2 ML2 Respondents Respondents ML2 Responses REQM SP 1.1-1 PP SP 1.ATLAS #4 – Interpretational Issues Which 3 ML2 specific practices are most likely to encounter interpretational issues? Interpretational Issues .2-1 CM SP 3.3-1 MA SP 1.ML2 Response Data by Rank PA Practice Practice Title Percent of Rank Cum.4-2 PP SP 2.

5-2 Comprehensive Methods RD SP 3. % of ML3 ML3 within ML3 Respondents Respondents ML3 Responses Establish Operational Concepts and Scenarios Establish Guidelines for DAR SP 1.1-1 Decision Analysis Evolve Operational Concepts TS SP 1.ML3 Response Data by Rank PA Practice Practice Title Number of Percentage of Rank Cum.1-1 12 12 11 11 10 16% 16% 15% 15% 14% 1 1 3 3 5 6% 12% 18% 24% 29% 35 .2-3 Package Validate Requirements with RD SP 3.ATLAS #4 – Interpretational Issues Which 3 ML3 specific practices are most likely to encounter interpretational issues? Interpretational Issues .2-2 and Scenarios Establish a Technical Data TS SP 2.

5-1 Establish Process Performance Models Select the Subprocesses that Will Be Statistically Managed Compose the Defined Process Establish Process Performance Baselines Select Processes 26 13 12 11 10 49% 25% 23% 21% 19% 1 2 3 4 5 19% 28% 37% 45% 36 52% QPM SP 1.ATLAS #4 – Interpretational Issues Which 3 ML4/5 specific practices are most likely to encounter interpretational issues? Interpretational Issues . % of ML3 ML3 within ML3 Respondents Respondents ML3 Responses OPP SP 1.4-1 OPP SP 1.ML4/5 Response Data by Rank PA Practice Practice Title Number of Percent of Rank Cum.2-1 OPP SP 1.3-1 QPM SP 1.1-1 .

8 GP 2.GP Response Data by Rank PA Practice Practice Title Number of Percent of Rank Cum.2 GP 2.7 Monitor and Control the Process Plan the Process Collect Improvement Information Objectively Evaluate Adherence Identify and Involve Relevant Stakeholders 20 12 11 9 7 26% 16% 14% 12% 9% 1 2 3 4 5 22% 36% 48% 58% 66% 37 .2 GP 3.ATLAS #4 – Interpretational Issues Which 1 Generic Practice is most likely to encounter interpretational issues? Interpretational Issues .9 GP 2. % of ML3 ML3 within ML3 Respondents Respondents ML3 Responses GP 2.

Conclusion The SEI’s lead appraiser upgrade training included a module on model interpretation issues. resolved and communicated. discussed. such issues need to be identified.” 38 . The conclusion drawn by the SEI Visiting Scientist that authored that section is: “Model interpretation issues will always exist. For the benefit of the lead appraiser community and that of our constituents.

otoole@att.net (And don’t hesitate to email suggestions for other “interesting” topics!) 39 .Questions? To be added to the ATLAS distribution list. send an mail to: Pat O’Toole PACT.