UNIVERSITEIT TWENTE. master stnesis tmpirical evaluation of change impact predictions using a requirements management too! with tormal relation types A quasi-experiment R.SA. van Vomburg Enschede, 25 November 2009 ‘Sbttware Engineering Group Faculty of Electrical Engneering, Mathematicsand Computer Science University ot Iwente Final project (29997) Business | nformation Technology ‘School of Management end Governance University ot Iwente graduation cormites dr. ir. K.G, van den Berg EW! dr.A.BJM.Wijnhoven MB dr. |. Kurtev EW! A. Goknil, MSc Ew!Acknowledgements First, | would like to thenk my graduation committee for their invdu- able advice and input. | have regarded our regular meetings as very enjoy- ableand beneficial to the qudity of my work Second, | would like to thenk al participants for their participation in the experiment and Johan Koolwaalj end Martin Wibbels for their expert Insight into the WASP requirements spedification. Your input has been very Important to attain eny research resuitsin the first place. Third, | would like to thank Ton Augustin, Pieter van Rossum, Klaas ‘SSkkel and Theo Thijssen for their expert insight into requirements trace- ability Your comments have enabled meto reflect upon this research from a practical perspective, Last but not least, | would like to thank Kim Scholte van Mast for her review of my find draft. Your comments have improved the correctness and readebility of thisthessAbstract Backg ound This research was part of a master's thes to evudte the impact of using TRIC, a software tool with formal requirements relation- ship types on the quaity of change impact prediction in softwere. (tj ative To andyze the real-world impact of using a softwere tool with forma requirements rdationship types, for the purpose of the evaluation of effectiveness of tools, with respect to the quaity of change impact predic- tions In the context of software requirements management; trom the view- point of system maintenance engineers Mabhod: This research feetures a quas-experiment with 21 masters de gree students predicting chenge impact for five chenge scenarios on a red- World software requirements specification. The quality of change impact predictions was measured by the F-measure and the time in seconds to complete the prediction ‘Two formal hypotheses were developed. Null hypothess 1 stated that tne F-scores of chenge impact predictions of system maintenance engineers us- ingT RIC will be equa to or lessthan those from system maintenance engi- neersnot using TRIC. Null hypothesis 2 sed that thetime taken to com- plete change impact predictions of system maintenance engineers using TRIC will be equa or longer than those from system maintenance engi- neers not using TRIC. The deta were collected by a web application and analyzed using ANOVA and !? datistica anayses Realts No significant difference in F-soores between TRIC and the other groups was detected. TRIC was found to be significantly dower for four out of five change impact predictions These inferences were made et " =0,05with amem statistica power of 54%. Limitations The vaidity was hampered by alimited avalability of usable software requirements specifications, experts from industry end theory re- garding the impact of change scenerios on change impact prediction. The results cannot be generalized for other software requirements specitica- tions, change scenarios or groupsof participants The condition to providea solution vaidation was therefore not met. Canduson: Empirical experiments cannot provide a solution validation to new software tools because there are not enough experts in the new tool Using TRIC to perform change impact prediction on a softwere require- ments specification of low complexity does not yield better quality predic- tionsbut doestake a longer time.lapie oF LOTILENLS 1. Introduction: 8 1.1. The QuadREAD Project. 13 1.2. Requirementsmetamodd. one ttn YS 1.3. Problem statement 16 1.4, Research objective! 16 1.5, Research method | a = - _— sca 1.6. Contributions: 19 1.7. Document structure! 19 2. Background and related work ! 24 21. Introduction. a 22. Software requirements... nnn sntsnnnnnnnnens 23. Software requirementsspeaifications | 24 24, Software requirementsmanagement!. 25 25, Sytem maintenance engineers !.nonminnennnnensnn inition 26. Change scenarios! 26 27. Changeimpad predictions! 2 28. Requirements models end relations... 32 29 Softwaretods:. 34 2.10, Validation approaches! er) 2.11, Conduson! 49 3. Experimental design : 51 34. Introductions. 51 32 Goal. 51 33. Hypothess: 5A 34, Design. 52 35. Parameters! 53 36. Variebles:. 5 37. Pranningt. 56 38 Partigpants!. 61 39. Objects 82 3.410, Instrumentation 1 64 3.11. Data collection. n 3.12, Analysisprocedurel.... wo 3.13. Validity evauation : 72 3.14. Conduson! 74 vii4. Execution! 75 A, irecCHOM Lsnnninntientnnitntntinasni easel 42. Sample. 75 43. Preparation 75 44, Data collection performed 1 TB 45. Validity procedures 78 46 Conduson! 79 5. Analysis! at 5.1. Introduction. 81 52. Change scenario representativenes... 8A 53. Golden sendard rdiabilty: 82 54, Precison-Recall and ROC graphs! 86 55. Oneway between-groups ANOVA... 86 56. Non-parametrictedings. 1 57. Analyssot coveriance! 94 58 Multivariate analyssof variance. 95 59 Conduson! %6 6. Interpretation! 97 64. Introduction. 97 62. Change scenario representativeness! 97 63. Golden sendard rdiabilty: 97 64. Precison-Recall and ROC graphs 99 65. Oneway between-groups ANOVA... 99 66. Non-paramatricteting: 99 67. Anayssot coveriancet. 100 68 Multivariate analyssof variance... snssnmnanninnnnsnnnnnsnnnanannnse tO 62. Conduson. 101 7. Conclusions and future work! 103 74, Summary... on sen tennis AB 72. Results: 104 7.3. Limitations:. 104 7.4, Futureworki... 106 8& Glossary! 109 9. References! 113 A. Interviews: 119 AA. Introduction 119 A2 Goat 19 viiiAB, AA AS. AS. AT, BA. B2 Ba B4 BS Be Bz. ca. C2 C3 ca. cs. Da. D2. D3 D4 Ds. Ds. E4. E2 E3 E4 ES, E6 FA F2 F3. Fa. Preparation | EX€QUION basen : Information systems academic. Industry expertsat Capgemini ‘Condusions.... Introduction : : 2 ‘Warming up (REQ_BDS_007)! Task 1 (REQ_PHN_001)! Tes 2 (REQ_SPM_004)... Task 3 (REQ_MAP_002)1. Task 4 (REQ_NAV_003)! Tak 5 REQ_TOR 001)! Group matching! Introduction ... Coding. Preexperiment randomized: Preexperiment tuned. Experiment final Golden standards! Introduction 1 Tes 1 (REQ_PHN_001 Task 2 (REQ_SPM_004)! Task 3 (REQ_MAP_002). Tes 4 (REQ_NAV_003), Task 5 REQ_TOR_001 Box plots! Introduction a Task 1 (REQ_PHN_001)! Task 2 (REQ_SPM_004)! Task 3 (REQ_MAP_002)... Task 4 (REQ_NAV_003)! Tak 5 (REQ_TOR 001)! Precision-Recall and ROC graphs! Introduction Legend: Tak 1! Tak 2! 119 120 120 122 125 127 127 127 127 127 127 128 128 129 129 129 130 131 132 133 133, 133 135 137 139 142 145 145 146 147 148 149 150 151 151 151 162 153,FS. FS. FI. GA Task 3! Tak 41. Tak 51 WASP requirements: Introduction 1. 154 155 158 187 187LISt Ul aVUTeEVIAaLiONS Als ANCOVA ANOVA Bs Ew! bis FPIS cul lec IEEE 1s0, MANOVA MB Moscow NR ° ome QuaiREAD Roc ad SL TBD TRIC UML URL wae Actua Impact Set Analysis of Covariance Analysis of Variance Estimated Impact Set Faculty of Electrical Enginesring, Mathematics end Computer Science Discovered Impact Sat Fase Positive Impact Set Graphical User Interface International Electrotechnical Commission Institute of Electrical and Electronics Eng nests Internationa Organization for Standardization Multivariate Andlysis of Variance ‘School of Management and Governance ‘Must have, Should have, Could have, Won't have Non-Randomized Experimenta observation Object Management Group (Quality-Driven Requirements Engineering and Architecture Design Reosiver Operating Characteridtic Standard ‘Systems Modeling Language To Be Determined Tool for Requirements Inference and Consistency Checking Unified Modeling Language Uniform Resource Locator Web Architectures or Services Platforms Experimenta treatment xi1. imtrouucuen This master’s thesis reports on the evauetion of the impact of using a software tool with for- ma requirements relationship types on the quality of chenge impact predictions in software, The tool and forma requirements rdationship typeswere developed as part of a requirements meemode in a research project called QuadREAD, which will be introduced before desorib- ing the problem statement, reseerch objective, context and further document structure. 1.1. Ihe QuadREAD Project This research is conducted at the laboratory of the Software Engineering Group from March 2009 up to and including November 2009. It tekes place within the context of the Quad- READ Project, which isajoint research project of the Software Engineering and Information ‘Systems research groups at the Department of Computer Science in the Faculty of Electrical Engineering, Methematics and Computer Science at the University of Twente. The Quad- READ Project runs from December 2006 up to and including December 2010. The context of the QuadREAD Project isthe erly phesesin software development processes: the etablishment of user requirements based on andysis of business goals and the application domain end the subsequent architecture design of desired systems The first phase concerns requirements engineering; the second, architectural design. In practice, it appears that these ‘two phases are poorly integral ed [50] The project ams abetter dignment between anaysts and architects The project elaborates on traceebility reseerch and focuses on tracing between user requirements and architectural design decisions [50]. Tracesbility isdefined asthe degree to which arelationship can be esteb- lished between two or more productsof the development process especialy products having a predecessor-successor or master-subordinate relationship to one another [58]. (One depiction of traceability in software development is constructed by combining two spe- idizetions of treceebility in the context of requirements engineering [64]. First, a distinction can be mae between pre-requirements specification tracecbility (forwerd to requirements and backwards from requirements) and post-requirements specification traceebility (forward from requirements and backwards to requirements) [25]. Second, intertevd and intratevel trace dependencies may be distinguished [3]. See Figure 1. 13Figuret:Tracebilityin s3' ware development [61] Figure 1 shows sever types of traceability. For exemple, requirements ements ere traced backwardsto dementsin business models and forward to dementsin thearchitecturd design. Requirements ements may heve intreJevel dependency relations end may evolve to anew configuration of requirements elements. There are traceability links between artifacts and links representing the evolution or incremental development of these artifacts[64 In a goa-oriented approach, the QuadREAD Project is developing a framework in which the dignment of requirements engineering and architecturd design is supported with practical Guidelines end tools The specific contribution of the project liesin the quantification of qud- ity attributes and tradeoffs in relation to trace information [50] The project will provide a framework for quditative and quantitative reasoning about re- quirements and architectural decisions to ensure sdected quality properties Thereby it will eneble decision-meking in the qudity-driven design of software architectures meeting user requirementsand system properties [50]. The research conducted in the QuadREAD Project is intended to have practical applicability by the centrd role of case studies from participeting business partnersiin the project, includ 4ing Atos Consulting, Chess Information Technology, Getronics PinkRoccade, Logica CMG, Shell Information Technology and Kwerds Consultancy [50] This research is part of the final project of a master's student of Business Information Tech- nology, which is a master’s degree program thet is headed by the School of Menagement end Governance of the Universty of Twente. The final project Is worth 3 Europeen Credits. It Is supervised by two assistent protessors, one from the School of Management end Governance and one from the Faculty of Electrical Engineering, Mathematics and Computer Scienos, as well 2s a postdoctoral scholar and Doc- tor of Philosophy student trom thelatter faculty. Biweekly meetings are held to evduete the research progress, qudity and results: Feedback Was aso provided by research fellows trom the I nformetion Systems Group end business pert- ners participating in the QuadREAD Project, as well as other master’s students executing ‘thelr Tindl project at the Software Engineering Group. 1.2. Requirements metamogel Research in the QuadREAD Project has contributed a requirements metamodel with formal requirements relationship typesto enable reasoning about requirements [29. Henceforth, this metamode will be referred to as thererpirementsmedaradd, It was constructed based on a re- view of literature on requirements modds The project dso contributed a prototype softwere tool named TRIC that supports the requirements metamoddl. TRIC was illustrated using a Single fictional case study featuring a course management system [37] Based on the case s.udy results it wes concluded that TRIC supports abetter understanding of mutud dependencies between requirements, but thet this result could not be generalized pending anumber of industria and academic case studies with empirical results [25 This research on the requirements metamodel can be classified astechnique-driven with alack of solution validation [69]. This classification does not imply that the research quality is poor: papers presenting new technology do not necessarily need to validate the proposed solution, ‘though they should explain why the solution, if validated, would be useful to stakeholders. Validation that @ proposed solution actually satisties the criteria from an analysis of stake- holder godsis@ research problem end doesnot need to be done in atechnology paper [70] 151.3. Problem statement The problem that this research deals with is the lack of solution vdidation of the require- ments metamodel, which can inhibit its adoption because the benefits are not dear It should be clear to practitioners for which problems a technique has shown to be successful in the real world [69] 1.4. Researcn objective The research objective should formulate ameansto providing a solution to the research prob- lem. Asa starting point, this paragraph compiles a set of solution requirements A research objectiveis subsequently formulated. solution requirements The research objective should work towards setisfying two solution requirements: 1. It should evduete the requirements metamodel as a real-world solution [69] on criteria thet were defined in its original research [70]. 2. It should be aigned with the gods of the QuadREAD Project, because that isthe context in which this research takesplace. The following peragraphs construct a research objective in en iterative fasnion by exemining these solution requirements more dosely. evaluation criteria in original research The origina research has defined two evaluation criteria for the requirementsmetanodel: 1. Thenumber of inconsistent rdationsin requirements documents 2. Thenumber of inferred new relations in requirements documents Henceforth, sf" were requirements patfication is used es a replacement term for requirements documentsin the context of software engineering, The term "softwere requirements specifica. tion” isdefined in the | EEE Standard Computer Dictionary [58]. It will prove to be useful dur- ing upcoming discussions on quality of software requirements specifications, for which the | EEE has well-known recommended practices[59} Requirements modeling of the case wes performed in two iteretions using the TRIC softwere tool, which has support for the forma relaionship types from the requirements metamoda, 16The firs iteration revealed a number of incongstencies in the software requirements specifi- cation. This enebled the researchers to correct these issues The second iteration reported zero detected incon stencies [25). In this case, using forma requirements relationship types led to ahigher degree of consistency of the software requirements specification In addition to improved consistency, both iterations dso reported a greater number of rela ‘tions than wes given initially, The additional reletions were interred by using formal require- ments relationship types end led to greater knowledge about the specific requirements in the softwere requirements specttication in the context of requirements modeling [25.. However, the validity of this condusion may be questioned. Because no tools other than TRIC were used, it could dso be concluded that requirements modeling became more effec- tive because any software tool was used, There Is no evidence that specifically the forma re quirements metamodel that TRIC supports increased the effectiveness of requirements mod- ding Finally, engineers should study red-world problems and try to design and study solutions for them [69]. Likewise, this research should andyze the red-world impact of the forma require- ments metamodel by using red-world softwere requirements specifications and using a red- world impact measure, Consequently, this reseerch should address this threat to validity by andyzing the red-wworld impact of the formal requirements metamodel by analyzing TRIC aongside other require- ments modding tools, which support other and less formal requirements metamoddls. Alignment with QuadREAD Project goals The requirements metamodel contributes to the QuadREAD Project by providing better ‘techniques for chenge impact analysis which is necessary for cost-effective softwere develop- ment [6]. It intends to do so by improving the precision of software requirements specifica tions. Current techniques are imprecise [25 which can reduos the quality of software require- ments specificationsin terms of ambiguity, modifiebility end traceability [52]. Of dl users of a software requirements specification, system mantenance engineers are the most concerned with chenge impact analysis. System maintenance engineers use the require- ments to understand the system and the relationships between its parts during requirements management [5 7Indeed, impact is usually associated with maintenance effort [61]. By identifying potential im- pacts before making a change, system maintenance engineers oan greatly reduce the risks of embarking on a costly change because the cost of unexpected problems generaly increases with the leteness of their discovery [1]. Having high-quality change impact predictionsis thus beneficia to gystem requirements management, Goal- Question- Metric approach ‘Subsequent to the considerations above, a research objective can be formulated according to the god template of the God-Quesion-Metric approach (79. The research objective can be formulated asfollows, To improve the adoption of the requirements metamodd and advance the state of the at in changeimpact andysis the research should! Analyze the red-world impact of using a software tool with formal requirements retation- ‘ship types; for the purpose of the evaluation of effectiveness of tools; with respect to the qudity of change Impact predictions; in the context of software requirements manage- ‘ment; from the viewpoint of system maintenance engineers. Operationalizations for thisgod are provided in Chapter 2 1.9. Kesearch metnod The reseerch method will involve performing change impact analysis on selected change sos- narios on softwere requirements specifications in two ways. using classic softwere tools end using the prototype TRIC software tool that supports forma requirements relationship types. ‘Such a research setup involves control over behaviord events during change impact enaysis, for which experimenta research isthe most appropriate [72] Experimental research has several subtypes, one of them being quasi-experimental research. By definition, quasi-experiments lack random assignment. Asignment to conditions is by ‘means of self-selection or administrator selection [52] such as isthe case in our setup with s8- lected chenge scenarios and a predetermined set of softwere tools. Consequently, quasi- experimentation is the most appropriate research method The quesi-experimental reseerch design is described in Chepter 3 181.0. Contributions Through a systematic design and execution of a quasi-experiment to empirically vaidate the impact of the TRIC software tool on change impact predictions, this research reveds the fol- lowing: + Empirica experiments cannot provide a solution vaidetion to new software tools becaise ‘there are not enough expertsin the new tool. This is aphenomenon thet will epply to ny research regarding new softwaretools. + Approximating the experts by training a group of non-experts is difficult to do reliably end hampersinternal validity to such apoint thet an empirical approach to solution validation is infeasible. + Using TRIC to perform change impact prediction on a software requirements specification of low complexity doesnot yield better qudity predictions but does take a longer time than compared to using Microsoft Excel or IBM Ration RequisitePro. + It is hypothesized thet TRIC isa more intelligent software tool and its benefits will only materialize given a sufficiently complex software requirements specification + There isalack of theory surrounding the nature of change sosnarios which posesa reliability issue to any research that deds with them. 1.f. Vocument structure This document amsto present the research in a rigorous structure. Such a structure makes it ing information [30]. The re- easier to locete relevent information and lowers the risk of mi search ispresented 2s follows: + Chapter 2: Background and related work clarifies how this research relates to existing Work, Including a description of softwere requirements specifications, specitic requirements, their quality criteria the requirementsmetamodd and dternetive solutions, + Chapter 3: Experimental design describes the outcome of the experiment planning phese, induding goals, hypotheses, parameters, veriables, design, participants, objects, in- strumentation, deta collection procedure, anayss procedure and evaluation of the vaidity. + Chapter 4: Execution describes each step in the production of the experiment, induding the sample, preparation, data collection performed end vdidity procedure, 19+ Chapter 5: Analysis summarizes the deta collected and the treat ment of the data, includ- ing descriptive aistics, data set reductions and hypothess testing + Chapter 6: Interpretation interprets the findings from the analysis including a evdua- tion of results and implications, limitations of the study inferences and lessons learned. + Chapter 7: Conclusions and future work presents a summary of the study induding Impact, limitations and tuture work. A glossary and list of referencesis presented afterwards.4. DacKYyIVUTIU anu relaleu WUIK 2.1, Introduction This chapter describes the releted work thet is relevent for this research. The reteted areas follow from the research objective, which is repeeted here: Analyze the real-world impact of using a software tool with formal requirements retation- ship types for the purpose of the evaluation of the effectiveness of tools with respect to the quaity of change impact predictions in the context of software requirements man- ‘agement from the viewpoint of system maintenance engineers. ‘A conceptual framework for background and relevent work can be developed by relating the keywords in this research objective. The nature of the reaionshipsis discussed in the follow- ing paragraphs. See Figure 2 S = cauienn ewian 4 ! repute Feiteon a} “ramet F aponaty —o] sonatas —— cages enressea or sapeny, t t | ae » a oy Figure2 Conomtual Samewerk for background and relevant work ‘The topics in Figure 2 are discussed in the following order. First, core topics to introduce the domain are discussed. T hese are softwere requirements, software requirements specifications, softwere requirements management and system maintenance engineers Discussed next are topics that provide specific instrumentetion to this research. These ere change scenarios, change impact predictions, requirements models end rdationships end asoftware tools Finaly, the topic of experiments is raised with a discusson of the investigated ‘approach, alternative validation methodsend rated experiments 2.2. Sottware requirements The term requirement is not used in a consistent way in the software industry [59. This re- search uses the definition provided by the| EEE Standard Computer Dictionery [58]: 11 A condition or capability needed by auser to solve a problem or achieve an objective, 2. Acondition or capability by a system or system component to satisfy a contract, standard, ‘specification, or other formally imposed documents 3A documented representation of a condition or capability asin or 2 Requirements are part of a software requirements speatication [83]. Knowledge about the characteristics of requirements is thus necessary to understand softwere requirements specifi- cations as a greater whole, Requirements oan differ in structure, contents and style. The following paragraphs describe related work on these characterizetions Requirements structure Requirements are often written in naturd language but may be written in a particular re- quirements specification language. When expressed in specification language, they may addi- tionally retain their description in natural language. Representation tools cen describe the ex- ternd behavior of arequirement in terms of some ebstract notion [58]. Note thet TRIC does ot describe externa behavior of requirements but the relationships between requirements, and thusisnot arepresentation tool. Requirements may be uniquely identified it they nave a unique name or reference number, which facilitetes forward traceebility They may facilitate backwerds traceebility if they explic- itly referencetheir sourcein earlier documents[ 2]. ‘Some requirements descriptions use the phrase “to be determined” or “TBD”. In thet case, the description can state the conditions causing this¢atus what mus be doneto eliminateit, ‘who isresponsble for the dimination and when it should be eliminated [59] Requirements can be ranked for importance or stability. Sebility can be expressed in terms of the number of expected chenges to any requirement based on experience of forthcoming 2events [53]. Importance can refer to the level of necessity or priority [35]. One widdy used technique for ranking importance or necessity is called MOSCOW, which defines*Must Have", “Should Have, “Could Have" and “Wont Have requirement” rankings [7]. Any other scale may be developed [8], one example being the “Essentid”, “Conditiona” and “Optional” scde that ispresented in IEEE Std 830-1998 [59]. Prioritiesare usualy used as weighting factor end can likewise be measured on any sce [3]. A highly common way to express requirements is using the feature requirement style [38]. Ex- ample requirements expressed using this style ere the following: Rit: The product shall be able to record that a room is occupied for repair in a specified period R2-The product shal be able to show and print a suggestion for stafing during the next ‘two weeks based on historical room occupation. The supplier shall specify the calculation details R&: The product shal be able to run in a mode where rooms are not booked by room number, but only by room type. Actua room alocetion isnot done until chectcin. R4: The product shal be ableto print out a sheet in which room location for each room booked under one stay. Note that the requirements are described in natural language end have a uniqueidentifier, and are not ranked or expressed in a specification language. Other styles for expressing require- ments are discussed later. Requirements contents Requirements can be dassified depending on the kind of condition or capability that they de- scribe The casstfication is not stenderdized, but it Is generally agreed that functional re- quirements specify a function that @ system or system component must be able to perform [58] and that non-functional requirements specity how wall the system should perform its in- tended functions[2]. Additional dasses of requirements can be found in the literature. For example, Lauesen [39] so discussesthe following + Datarequirements: daathat the system should input, output and storeinternally. + Other deliverables: required ddiverebles besides hardware and software, such as docu- mentation and specified services 23+ Managerial requirements: when deliverebleswill be delivered, the price and when to pay it, how to check that everything isworking, what happensif things go wrong, etc. |EEE Std 830-1998 [53] dso recognizes these, but maintains that these should not be provided as spe- cific requirements but rather asa separate pert in software requirements specifications. Sommerville [55] dso discerns domain requirements that come from the application domain. of the system and reflect characteristics and constraints of that domain. These requirements may be either functional or non-functiond end thus are not truly a separate class of require- ments with respect to their contents For this reason, this reseerch disregerds domain re- quirements as a seperate classiticetion, Requirements styles Requirements may be expressed in a variety of styles depending on the dassification of a re- quirement. Lauesen [29] describes over 25 styles, including the previously illustrated feeture list style, Each style has its own advantages and disadvantages Indeed, there isno best styleto express requirements. TRI C only supportsthe featurelist style, 2.3. doTlware requirements specitications ‘Software requirements specifications are documentation of the essentid requirements of the softwere end its externd interfaces [12]. Documented representations of specific requirements in various styles are but one part of it, asit typically aso contansother dements[S5. The parts of software requirements specifications are not standardized, although several guidelines exis, induding | EEE Std 820-1998 [59], the Volere template [51] and those provided by Lauesen [32] and Sommerville (55, This research uses IEEE Std 820-1998 as leading guideline for two reasons. First, because it contains recognized quality criteria for software requirements specifications tha may serve as Useful metrics. Second, because it is igned with ISO/IEC 12207 [29], an industria standard in informetion technology for for softwere life cycle processes, which is useful in the context of change impact andyssand the QuadREAD Project, IEEE Sd 830-1998 discusses essentid parts of a software requirements specification and pro- vides several example templates on an informative basis [58]. The essential partsare ceptured In aprototype sotware requirements specitication outline, See Figure & 24Table of Contents 1. Introduction Purpose ‘Scope Definitions, acronyms and abbreviations References ae one Overview 2 Overall description Product perspective Product functions 1 2 3. User characteristics 4. Constraints 5 Assumptionsand dependencies. 3. Qesificrequirements Appendixes: Index Figure3 Protetypess' warerequirementsspeificatcn autline[ 2] Other guidelines generally agree with the parts thet a software requirements specification should contain. Differences lie in the ordering end composttion of parts. For example, the Volere templete dictatesto have seperete parts for functional end non-functiond requirements [54] while | EEE Std 830-1898 mekes no such distinction in its description of specific require- ments [58]. In ll guidelines, the parts containing requirements representations are seperate from perts containing domain and product insights. 2.4. sottware requirements management Requirements evolution both during the requirements engineering process and after a system has gone into service is inevitable. Software requirements menagement is the process of un- derstanding and controlling changesto requirements for software products (59, Requirements management should be done by a change control board with the authority to decide on chengesto be made or not. The basic change cycte is 2s follows [2] 11 Reporting: arequirementsissue is reported to the change control board 2. Analysis: theissueisanayzed together with other issues & Decision: evaluete the issue and plan what to do with it 254. Reply: report the decision to the source and other peopleimpacted by it & Carry out the decision: executethe plan This reseerch isinterested in the qudity of change impact predictions These predictionsarea result of change impact endyssin the and ysis phase 2.9. system maintenance engineers Chenge impact enaysis is performed by system meintenence engineers, which are a particular type of requirements engineer. System maintenance engineers use the requirementsto under- stand the system and the relationships between its pats [59]. Based on this understanding, they predict the impact that a requested change in @ particular requirement will have on other requirements. Increased understanding about @ software requirements specifications helps them to perform this activity effectively [25, 2.6. Change scenarios Requested changes can teke the form of change scenarios, which describe possible change Stuetions thet will cause the maintenance organizetion to perform changes in the softwere andlor hardwere[ 11] ‘Scenarios should define very concrete stustions. They may be assigned an associated weight or probability of occurrence within a certain time. For exemple, a change scenerio could be: ‘Due to anew type ot pump, the pump intertace must be chenged from duty cycle into a digl- td interface, with ase vduein KP (kilo Pascal)” [1] ‘Several soenerio-based methods have been proposed to evduate softwere architectures with respect do desired quality attributes such as maintainability, performance, and so on [8]. Asa systematic literature review with the query (‘change sumario’ OR “change smarios) AND so" - ware on the Scopus and Web of Science detabases turned out, there has been little focus on change scenariosthemselves. Generally, change scenarios may be dicited by interviewing stakeholders, Here, it isimportent to interview different stakeholders to capture scenarios from different perspectives. This adds to the diversity of change scenarios It is dso observed thet engineers have a certain bias in proposing scenarios that have dready been considered in the design of the system [38]. One downsideto diciting change scenarios from stakeholders is thet most suggested scenarios relate to issues very dose in time, eg. anticipated chenges To address this issue, it may be 26helpful to have some organizing principle while aliciting scenarios This principle may take the form of a possibly hierarchical, classficetion of scenariosto draw from [38]. Evduating sosnariosis hard. Ripple effects are hard to identify since they are the result of de- tailsnot yet known at the level in which the scenarios are expressed [38]. Indeed, architecture detailsare not known a therequirements level in this research In summary, theres little theory on chenge scenarios. That end problems regerding their rep- resentativeness and validity pose week nesses to methodologies that depend on them [11] 2.f. Gnange impact preaictions Change impact predictions enumerate the set of objects eximated to be affected by the change impect analysis method. Change impact enaysisisthe identificetion of potential con- sequences of a change, or estimating what needsto be modified to accomplish a change [6] ‘A number of sets can be recognized in the context of chenge impact prediction [2]. See Teble a ‘Abbre- Set viation Description System - | Set of all objects under consideration. Estimated Impact Set EIS | Set of objectsthat ae estimated to be affected by the change. Actual Impact Set AIS | Set of objects that were actually modified as the result of performing the change. False Positive Impact Set | FPIS | Set of objects that were estimated by the change impact andyss to be affected, but were not d- fected during performing the change. Discovered Impact Set | DIS | Set of objects that were not estimated by the change Impact analysis to be affected, but were fected during performing the change. ‘Table Changeimpad prediicn sds{2] Table 5 shows that the Estimated Impact Set, which isthe change impact prediction, may not be equa to theActua Impact Set. See Figure 11 ar‘System EIS Als. Figure'tt: Changeimpad pretidion sts{6] The Venn diagram in Figure 11 gives a visual representation of the chenge impact prediction setsin Table 5 In particular, change impect predictions may fsa estimate objectsto chenge (Fdse Postive Impact Set) or fasdy estimete objects to not change (Discovered Impact Set). Thisleadsto the thought thet there is aquality attributeto chenge impact predictions quality of cnange Impact predictions The extent to which the Estimeted Impact Set equals the Actual Impact S& is an indicetion of change impact prediction quality An object etimeted to change may indeed change or it may not; an object actualy changed may have been estimated or it may not have been, This may be ceptured using abinary classifier; see the so-called canfugon matrix in Table 6 [20]. ‘Actual Impact Changed Not changed Eatnet Changed True Positive False Posttive Impact Not changed False Negative True Negative ‘Table6: Confusion matrix [20] Binary Gassifiers are also used in the domain of information retrieval. Metrics from this do- main may be used to measure the quality of chenge impact predictions[J. See Table 7 28Metric Equation ‘Also known as. Recall Els! alg Hit rete, sensitivity, true postive rate (Als, Precision Els! alg Positive predictive value lEIs Fallout |FPIS, Fase darm rate, false positive rate {System !/AlS ‘Table7: Changelmpad pradicicn quality matrics{2] ‘A populer measure that combines precision and recall isthe weighted harmonic meen of pre- ison and recall, aso known as the F; measure because recall and precision are evenly waighted [2]. See Equation 1 _2 !precision recall precision + recall F Equation ‘:F; measuire{2] Measures such as Fosand F2 weigh either the precision or recall double end can be used if ei- ther precision or recall is more important than the other in a certan stuation (4. The F- measure is used the most and is henceforth referred to as the F-mexure. Results on the F- measure are referred to as F-sures Another quality attribute of change impact predictionsisthe effort that it takes. While the F- meesure can be regerded as a quality measure of chenge impact prediction products, the measurement of change impact prediction process effort is left to human judgement [12]. Time is one plausible metric [44] to measure effort but does not represent it fully For exemple, a group using TRIC may take much longer looking at visualizetion output, while viewing the visualization may take only one mouse-cick. Visualization tecnniques Popular methods to visudize measurements on these metrics are the Receiver Operating Cheracteristic, or ROC curve, and Precision-Recdl graph [2]. The ROC curve is a grephical Plot of the recal versus falout. The Precision-Recal graph is exactly thet: a graphica plot of the precision versus recall. See Figure end Figure 18 29Jet Operating Characteristic Figuret2 Rexive Opaating Charadaisic Figure 13 shows a ROC aurve of change impact predictions made by different people for the seme change scenario. The X axis displays their fallout scores on a scale of O (no Taise pos- tives) to 1 (ll possible false positives). The Y axis displays their recall scores on a sede of 0 (no ‘true positives) to 1 (al posable true postives). The circies represent the scores of the individ- a predictions In aROC curvethe black diagonal line from (0, 0) through (1, 1)is called thelinedt nodisrimi- rtion [2 Scores along this line are effectively random guesses the estimetions were com- prised of an equa number of true postivesand fase postives. The diagonal lines thet are pardilel to it areisocost lines. Soores dong these lines have an equal cost of fase negatives versus false postives. It is thus desirable to maximize recall and mini- mize falout, placing scores as far away as posable from the line of no discrimination in northwestern direction & a 90° ange [J ‘Scores southeast of the line of no discrimination are called pavers stres becaise they are worse than those of random predictions. Such scores may be transformed into better-than- random scores by inverting their Estimeted Impact Sets, effectively mirroring the score over theline of no discrimination [2] 30The gradient of the isocost linesin this ROC curve are at 45; indicating that the cost of fdse negatives versus fase posttives is equa, which is a common assumption. Like the F;-soore, an ‘emphasis may be placed on either false negatives or fase positivesif appropriate to the stua- tion, in which case the gradient will chenge [2] Precision- Recall Graph Recall Precision Figure te: PredsinRealgraph Figure ‘B shows a Precison-Recall greph of change impact predictions made by different peo- ple for the same change soenario, The X axis digplaystheir recall scoreson ascde of 0 (no true postives) to 1 (al possible true postives). The Y axis displays their precision scores on a sae of 0 (estimation conteined no true positives) to 1 (estimetion conteined only true posttives). ‘The cirdesrepresent the scoresof the individua predictions The isometric lines show boundaries of F-scores, from southwest to northeast: 0,2; 0.4; 0.6 and 08 It isthusdesrableto maximize both precision end recall [2] ROC curves are commonly used to present results for binary decision problems. However, when deding with highly sewed datasets, Precision-Recall graphs give a more informative picture [13 312.8. Requirements models and relations One requirement in a software requirements specification may be related to one or more other requirements in thet specification. Redlionships can be of acertain type that more pre- cisely defines how the requirements are rated. Using imprecise relationship types may pro- duce deficient resultsin requirements engineering. For exemple, during change impact andysis requirements engineers may have to menudly endyze all requirements in a softwere require- ments specification. This will lead to more costly chengeimplement ation [25] Different vocebularies with types of relationships exist. For example, IBM Rational Requi- stePro defines tracsTo and traaFran [27]. These only indicate the direction in the relationship and are thus very generic [29. Another example is OMG SysViL, which defines amntain, copy, daive [47] end includes the senderd refine UML stereotype [67]. These are defined only in- formally in natural language and therefore imprecise. The QuadREAD Project has contributed a requirements metamodel with forma relationship types see Figure 4. The formdizetion is based on first-order logic and is used for consistency checking of relationships and inferencing. Consistency checking isthe activity to identify the relationship whose existence causes a contradiction. Inferencing isthe activity of deriving new relationships based solely on the relationships which a requirements engineer has dready specified [29 Fequremenavocel] hare Sing (* [ [ent Sn, base tara SE == Fines == (Genres) [Gontains Figured: Requirenentsmeamede [25]In Figure 4, a software requirement s specification is composed of eny number of requirements and relationships Each relationships has one of five types according to the following informal definitions [29] and illustrations [24 + Requires relationship. A requirement Ri requires a requirement Ro if Ri is fulfilled only when Re is fulfilled. The requirement can be seen asa precondition for the requiring re- quirement (69. See Figure 5 a JH cures +f Figures Requiresrd aticnship [24] + Refines relationship. A requirement R: refinesa requirement Ro if Ri Is derived from Roby ‘adding more details to its properties The refined requirement can be seen as an abstraction of the detailed requirements| 14] [62]. See Figure 6, Figure: Reinesrdationship[24] + Partially refines relationship. A requirement R; patie¥lrdinesa requirement Ro if Riis derived from Re by aiding more details to parts of Re end exduding the unrefined parts of Ro, This relationship om be described as a special combination of decomposition end re- finement (62). SeeFigure 7, ‘a = . . Figure? Partid Grdinesr tionship [24] + Contains relationship. A requirement Rj oontains requirements Ro..Rn if Ro..Re are parts of the whole R; (part-whole hierarchy). This relationship enables a complex requirement to be decomposed into parts[47]. See Figures 8 and 9. 33Me conta 7 oas 4 = Figure8 Containsrdaienship partial dexampastion [24] contains -» fi] Fe RI Figure9, Cantainsréaticnship: campletedexmpasition [24] + Conflicts relationship. A requirement Ri cnilidswith a requirement Ro if the fulfillment of Ri excludesthe fulfillment of Ro end vice versa [6q. There may be conflicts among multi- ple requirements thet are non-conflicting parwise. See Figure 1, 1 J cones + Figure 1: Cenficsedationship[ 24] The requirements metamodel aso has formal definitions of these relationships end proofs for ‘the consistency checking and interencing capabilities. They are omitted here because the in- forma definitions convey enough information to a practitioner to apply them in software re- quirements management. This was confirmed in the QuadREAD Advisory Board Meaing with the project partnerson une 4, 2009. 2.9. soTtware tools This research investigates three softwere tools that support requirements menagement @ dlit- ferent levels of intelligence and maturity + Microsoft Exod Is@ popular general-purpose spreadsheet epplication. + IBM Rational RequisitePro is a dediceted requirements management application that is wall-known in the industry 34+ TRIC is a prototype requirements management tool with support for the forma require- ments relationship types The following paragraphs discuss feetures of these softwere tools, present a dassification scheme for comparison end findly compare the softwere tools. Features Requirements management tools may support many features and not all will apply to this re- search. The TRIC feeture list containsthe following [29 + Management of requirements: the creetion, updating, viewing and deletion of requirements. The softwaretool should support thisin agraphicd fashion for ease of use, + Management of requirements relationships: the creation, updating, viewing and deletion of relaiions between requirements. This effectively adds traceebility support to the software tool, which hasbeen shown to be important to practicing effective chenge management. + Traceability matrix visualization: the display of related requirementsin an n-n matrix. Thisis ‘a. common wey to visualize traceebility in a softwere requirements specification and may be used to propagete changes from one requirement to related requirements, which is useful in change impact enalysis[]. + Automated reasoning based on requirements rdaionships, such as + Displaying inconsistencies the automated detection end visualizetion of inconsistencies in the requirements. This will only be possible if the softweretool supports management of requirements rétionshipsend their relationship types carry semantics. + Displaying inferred rdletionships the automatic detection and visualization of require- ments relationshipsthat were determined to exist based on given requirements relation- ships In its simples form, this may be done by applying transitivity. More edvenced tools.cen apply more advanced reasoning if their relationship types carry semantics + Explaining reasoning results: the visualization of the process of consistency checking end inferencing. This provides additiond knowledge while practicing chenge management, which can make it more effective. 35Microsott Excel Microsoft Exo Is @ popular genera-purpose spreadsheet epplication. Although it Is not @ dedicated requirements management tool, it can be used to keep allist of requirements and relate them to each other, for example using atraceability matrix. See Figure #4. Figure 4: Tracesility matrix inMicoss' Exed Because Microsoft Excel is not a dedicated requirements management tool, performing re- quirements management with it will largely be an ad-hoc activity. It carries no semantics and cannot perform interencing or consistency checking This research uses Microsoft Exca 2003, IBM Rational Kequisitebro Rétlona Requisterro is a requirements management end use-case writing tool, intended to help improve communication and traceebility, enhance collboretive development, and inte- gréte requirementsthroughout the litecycte [27]. This research uses | BM Retional RequisitePro version 71. As.a mature requirements manege- ment tool, it offers many fectures. The following fectures are of interest to this research re- garding traceability and chengemenagement:+ Managing requirements. Requirements can be crested, updated, viewed and deleted us- ing aGUI. See Figure 18 lesb eweweamel gl Figure 1S GUI for managingrrequirementsand rdationsin | BM Retiona RequistePro + Managing requirements relationships. Relationships oan be created, updated, viewed and deleted using the requirement menagement GUI in Figure 150r the traceebility matrix in Figure %6. Figure 16:Tracesility matrix inI BM Rational Requiste?ro 37+ Displaying suspect relations. Relationships deemed suspect based on transitivity are highlighted. In Figure 16, ardationship issuspected between A 25and ASD. TRIG A prototype software tool was developed for requirements modeling with support for the formal requirements relationships thet were defined in the requirements metamodel. This software tool is caled TRIC: Tool for Requirements Inferencing and Consistency Checking (64). This research uses TRIC version 1.10. It supportsthe following features[25, + Managing requirements. Requirements cen be creeted, updated, viewed and deleted us- ing aGUI. See Figure 17 [@ mecnerovenreooetree te sentehnnetecevoon, Figure GUI fer managingrequirenentsandrd aticnsin TRI + Managing requirements relationships. Relationships can be created, updated, viewed and deleted using the requirement management GUI in Figure 17 or the traceability metrix. in Figure *8.pooooooooCoSooso0IoC' gpoooooooooooosqoooooc {poooooooooooosocoooor gpoooooposooososososooc gpoooooosconsosososoor g PooooopocoooosCoooooc poooooooogosssecosaoc gpooonosoooooeooooooc gboooooooocooooosoooooc #Pooooooooosoososooooc gpooonopooonsoeoensooc ooononoocossnssossoc pooooopoooooosooooonr| gpooooooooooooeqoooonr| spoooooosooososococoor #pooooooooosoosoooooor| japonABaaasAcnasoaanc| poooooooocossooco090c, #poooooooooooooooooo0r| spooonoosoooooeqoooooc gpooooooooooooo0ooo0o0oC ql alalelslsl2i3 7 Figure 18 Traaebility matrix in TRIC + Displaying inconsistencies and inferred relations. Inferred relationships are high- lighted and alist of conflicting requirementsisprovided. See Figure 19 and Figure 20. Figure-9: Output cf theinferendng activity in TRIC. 39ots Figure20: Output d theaansstenoy chekking adivityin TRIC. + Displaying results of reasoning. Sets of requirements and their relations may be ex- pressed es a graph [19]. The given and inferred relationships between requirements are visu- alized for requirements engineersto interpret. See Figure 21