Obtaining Well-Founded Practices about Elicitation Techniques by Means of an Update of a Previous Systematic Review

Oscar Dieste, Marta López, Felicidad Ramos, Fraunhofer IESE Complutensian University of Madrid Indra Systems

either confirming or refuting the previous findings . the conclusions of earlier SRs should be updated.Systematic reviews (SR) Outline Overview Instantiation Performed Methodogy Findings Discussion Conclusions References  Goal:  Pool together the results obtained in different empirical studies and propose recommendations based on the best available evidence Amount of evidence available at the time SR is done Need for updating:   Critical factors:   If new empirical studies are discovered (because they were not identified before) or carried out.

Our SR instantiation Outline Overview Instantiation Performed Methodogy Findings Discussion Conclusions References  SR on Requirements Engineering field Elicitation techniques selection  Relevance:  Techniques applied to extract knowledge from the requirements stakeholders  Critical for improving the communication process among software engineers and stakeholders  .

SRs performed Outline Overview Instantiation Performed Methodogy Findings Discussion Conclusions References  Previous SR: 53 publications concerning individual elicitation techniques 26 selected publications (Previous SR review) 27 unavailable publications (initial search) .

(grey literature) 4 useless public. (no empirical studies) 9 useful public.SRs performed Outline Overview Instantiation Performed Methodogy Findings Discussion Conclusions References  Updating SR: 53 publications concerning individual elicitation techniques 26 selected publications (Previous SR review) 27 unavailable publications (initial search) 14 unavailable public. (Update SR review) .

Aggregation 10. Response variables analysis 6. Training on SR 1. Final results decodification . Treatment analysis 5. Setting up supporting tables 4. Generalization Pending treatments/ response variables? No Yes 8. Primary studies reading and data extraction 3. New evidence extraction 9. Selection of primary studies 2.SR Methodology Outline Overview Instantiation Performed Methodogy Findings Discussion Conclusions References 0.Development of results table 7.

2006 . see   Dieste. as well as with the results of previous SR." http://grise.php. vol." IEEE Transactions on Software Engineering. Agregación de las evidencias obtenidas de los estudios empíricos relevantes. and Juristo. O. N.es/research_documents. "Systematic Review and Aggregation of Empirical Studies on Elicitation Techniques. O.upm."TR5. evidences which can be later used to identify in which situations a given elicitation technique is useful can obtained  Aggregation process. 2008 Dieste.SR Main Findings Outline Overview Instantiation Performed Methodogy Findings Discussion Conclusions References   After performing the SR.. more than 60 new empirical results were obtained Combining those empirical results among themselves.

28] [5.9.29] 21 33 36 37 38 39 40 REINFORCES REINFORCES NEW NEW NEW NEW NEW [1. like protocol analysis.9] Support [28] Neutral [29.29] [29] . than for unstructured interviews or vice versa Transcription time cannot be established as being longer for sorting techniques than for laddering or vice versa Laddering gathers fuller information than sorting techniques The efficiency of unstructured interviews is greater than scaling techniques Laddering and scaling techniques have the same efficiency Scaling techniques are more difficult to apply than unstructured interviews Laddering is more difficult to apply than unstructured interviews Laddering and scaling techniques have the same difficulty [1.28] [5.30] Opposes [5] ID 12 16 REDUCES [1] [28.SR Main Findings Outline Overview Instantiation Performed Methodogy Findings Discussion Conclusions References  After performing the SR.28. more than 60 new empirical results were obtained KEY REFUTES Result of the aggregation There do not appear to be any differences in terms of session duration between unstructured interviews and laddering (this evidence is not longer valid Transcription time cannot be established as being longer for introspective techniques.9] [5.9] [5.9] [5.

has been identified Reduces   Reinforces   New  . but reduces our confidence in it A previous evidence is supported with new compatible results A new evidence. not present in the existing set.SR Main Findings Outline Overview Instantiation Performed Methodogy Findings Discussion Conclusions References  Types of findings:   Refutes  The newly gathered results refutes a previous evidence The new results cannot refute a previous evidence.

although some contrived techniques like laddering or card sorting were equally effective in some cases. The current work strengthens this conclusion.  Effectiveness and efficiency:  In the previous SR.29] Opposes  Time-consuming:  ID 16 KEY REDUCES .Discussion Outline Overview Instantiation Performed Methodogy Findings Discussion Conclusions References  Focusing on interviews. Evidence 16 supports that fact (study suggest that interviews take more time than protocol analysis) Result of the aggregation Transcription time cannot be established as being longer for introspective techniques. it was found that interviews were overall the most effective elicitation technique. than for unstructured interviews or vice versa Support [1] Neutral [28. No empirical result contesting such effectiveness has been identified Interviews may be more time-consuming than other morefocused techniques like sorting or protocol analysis. like protocol analysis.

Conclusions Outline Overview Instantiation Performed Methodogy Findings Discussion Conclusions References  Updating Systematic Review  Identifying well-founded practices when selecting an elicitation requirements technique Interviews are in average the most effective elicitation technique Combine findings from empirical and theoretical works and expect opinion to develop a comprehensive theory concerning the application of elicitation techniques  Results:   Future work:  .

. M." Marketing Letters. and Shook. C. pp. L... USA. Rugg. "Knowledge acquisition methodologies: Survey and empirical assessment. 298-305. A. 1988. 174-182.. and Hedgecock. "Knowledge acquisition techniques for requirements engineering. . 153-165. V. P. 1994. B." Proceedings of the 8th Conference in Artificial Intelligence ECAI-88. R. [6] Bradburn. pp. and Raj. MN. vol. 1994. [4] Wood. S. pp. [5] Brandt. vol. vol. E. R. A. S. pp. N. 14. 8." Proceedings of the Workshop on Requirements Elicitation for System Specification. A. J. [7] Maiden. Keele." Behaviour & Information Technology . S. 8. 2005. "Attribute elicitation in marketing research: A comparison of three procedures. W. pp. UK.. Ford. "An exploratory study of two KA methods. pp. 11.. . "A comparison of knowledge elicitation methods. G.. J." Proceedings of the Ninth International Conference on Information Systems. [9] Steenkamp." Wood and Fiber Science. C. M. Davis. S. and Van Trijp.-J. 47-54. "Attribute elicitation: Implications in the research context." International Conference on Engineering Design (ICED'91). G. Shadbolt. and Lammersen. "Knowledge elicitation techniques in classification domains.. 1995. 1-23. 37." International Journal of Expert Systems.SR References Outline Overview Instantiation Performed Methodogy Findings Discussion Conclusions References  References used in the updating work [1] Burton. "Impact of cognitive abilities of experts on the effectiveness of elicited knowledge. 1991.. 127-146. vol. M. Minneapolis. P. and Rugg. vol.. 1995. J. L. "Evaluation of interviewing methods and mediating representations for knowledge acquisition. C. T. Clay. C. 77-87. and Salvendy. G. [3] Holsapple. N. H. [8] Grabowski. M. pp." Expert Systems. M... 1988.. [2] Chao. E.-B.. M. 1997..