Lecture 4: Requirements Elicitation Elicitation Techniques: University of Toronto University of Toronto

University of Toronto Department of Computer Science University of Toronto Department of Computer Science
Lecture 4: Requirements Elicitation Elicitation Techniques

Last
Last Week:
Week:  Traditional techniques  Cognitive techniques
Initiating
Initiating an
an RE
RE process:
process:  Introspection  Task analysis
Stakeholders
Stakeholdersand
andBoundaries
Boundaries  Reading existing documents  Protocol analysis
Goals
Goalsand
andScenarios
Scenarios  Analyzing hard data  Knowledge Acquisition Techniques
Feasibility
Feasibility andRisk
and Risk Card Sorting
 Interviews
Open-ended Laddering
Structured Repertory Grids
This
This Week:
Week: Proximity Scaling Techniques
 Surveys / Questionnaires
Elicitation
Elicitation Techniques
Techniques  Meetings  Contextual approaches
Interviews,
Interviews, Questionnaires,
Questionnaires, etc.
etc.
Cognitive  Collaborative techniques  Ethnographic techniques
Cognitive approaches
approaches Participant Observation
Contextual approaches  Group techniques
Contextual approaches Focus Groups
Enthnomethodology
Ethnography  Discourse Analysis
Ethnography asas an
an RE
RE technique
technique Brainstorming
Conversation Analysis
 JAD/RAD workshops Speech Act Analysis
 Prototyping  Sociotechnical Methods
Next
Next Week:
Week:  Participatory Design Soft Systems Analysis
Modeling
Modelingand
andAnalysis
Analysis(I)
(I)
Intro
IntrototoRE
REModeling
Modeling
Modeling Organisations
Modeling Organisations
Modeling
Modelingsoft
softsystems
systems
© 2000-2004, Steve Easterbrook 1 © 2000-2004, Steve Easterbrook 2
Background Reading “Hard Data” Collection

 Sources of information:  Identify Collections of Hard Data
 company reports, organization charts, policy manuals, job descriptions,  Facts and figures, financial information,…
reports, documentation of existing systems, etc.  Reports used for decision making,…
 Survey results, marketing data,…
 Advantages:
 Helps the analyst to get an understanding of the organization before  Sampling
meeting the people who work there.  Sampling used to select representative set from a population
 Helps to prepare for other types of fact finding  Purposive Sampling - choose the parts you think are relevant without worrying
 e.g. by being aware of the business objectives of the organization. about statistical issues
 Simple Random Sampling - choose every kth element
 may tell you the detailed requirements for the current system.
 Stratified Random Sampling - identify strata and sample each
 Clustered Random Sampling - choose a representative subpopulation and sample it
 Disadvantages:  Sample Size is important
 written documents often do not match up to reality.  balance between cost of data collection/analysis and required significance
 Can be long-winded with much irrelevant detail  Process:
 Decide what data should be collected - e.g. banking transactions
 Appropriate for  Determine the population to be sampled - e.g. all transactions at 5 local
branches over one week
 Whenever you are unfamiliar with the organization being investigated.  Choose type of sample - e.g. simple random sampling
 Choose sample size - e.g. every 10th transaction
1
Example of Interviews
hard data
 Types:
 Questions:  Structured - agenda of fairly open questions
 Open-ended - no pre-set agenda
What does this data tell
you?  Advantages
What would you do with  Rich collection of information
this data?  Good for uncovering opinions, feelings, goals, as well as hard facts
 Can probe in depth, & adapt followup questions to what the person tells you
 Disadvantages
 Large amount of qualitative data can be hard to analyze
 Hard to compare different respondents
 Interviewing is a difficult skill to master
 Watch for
 Unanswerable questions (“how do you tie your shoelaces?”)
 Tacit knowledge (and post-hoc rationalizations)
 Removal from context
 Interviewer’s attitude may cause bias (e.g. variable attentiveness)
Source: Adapted from Goguen and Linde, 1993, p154.
Interviewing Tips Surveys and Questionnaires

 Starting off…  Advantages
 Begin the interview with an innocuous topic to set people at ease  Can quickly collect info from large numbers of people
 e.g. the weather, the score in last night’s hockey game  Can be administered remotely
 e.g. comment on an object on the person’s desk: “My,… what a beautiful
photograph! Did you take that?”
 Can collect attitudes, beliefs, characteristics
 Ask if you can record the interview  Disadvantages

 but put tape recorder in front of person  Simplistic (presupposed) categories provide very little context
 say that they can turn it off any time.  No room for users to convey their real needs
 Ask easy questions first  Watch for:

 perhaps personal information  Bias in sample selection
 e.g. “How long have you worked in your present position?”
 Bias in self-selecting respondents
 Follow up interesting leads  Small sample size (lack of statistical significance)
 E.g. if you hear something that indicates your plan of action may be wrong,  Open ended questions (very hard to analyze!)
 e.g.,“Could we pursue what you just said a little further?”  Leading questions ( “have you stopped beating your wife?”)
 Appropriation (“What is this a picture of?”)
 Ask open-ended questions last  Ambiguous questions (I.e. not everyone is answering the same question)
 e.g. “Is there anything else you would like to add?”
Questionnaires MUST be prototyped and tested!
2
Survey Sources of Error Meetings

 Sampling Error & Sampling Bias  Used for summarization and feedback
 Info not collected from every member of the target population  E.g. meet with stakeholders towards the end of each stage:
 Potential Bias in how the population is sampled  to discuss the results of the information gathering stage
 E.g. web-based surveys exclude non-internet users  to conclude on a set of requirements
 E.g. Respondents who self-select might not be representative  to agree on a design etc.
 Use the meeting to confirm what has been learned, talk about findings
 Coverage Error
 Occurs when not all the target population is included in sampling
 Meetings are an important managerial tool
 E.g. Use of an existing mailing list that is out of date  Used to move a system development project forward.
 Need to determine objectives for the meeting:
 Measurement Error  E.g. presentation, problem solving, conflict resolution, progress analysis,
gathering and merging of facts, training, planning,...
 Occurs when responses are not counted accurately
 E.g. ambiguous question leads to responses that are hard to classify  Plan the meeting carefully:
 E.g. question topics do not correspond to the issues being investigated  Schedule the meeting and arrange for facilities
 Prepare an agenda and distribute it well in advance
 Non-Response Error 

Keep track of time and agenda during the meeting
Follow up with a written summary to be distributed to meeting participants
 Occurs when significant number of population do not participate  Special rules apply for formal presentations, walkthroughs, brainstorming, etc.
 E.g. a significant special interest group refuses to take part
Group Elicitation Techniques Joint/Rapid Application Development

 JAD & RAD Principles:
 Types:
 Group Dynamics - use workshops instead of interviews
 Focus Groups
 Visual Aids
 Brainstorming  Lots of visualization media, e.g. wallcharts, large monitors, graphical interfaces
 Organized, Rational Process
 Advantages  Techniques such as brainstorming and top-down analysis
 More natural interaction between people than formal interview  WYSIWYG Documentation Approach
 each JAD session results in a document which is easy to understand and is
 Can gauge reaction to stimulus materials (e.g. mock-ups, storyboards, etc)
created and agreed upon during the session
 Disadvantages  Notes:
 May create unnatural groups (uncomfortable for participants)
 Choose workshop participants carefully
 Danger of Groupthink  they should be the best people possible representing various stakeholder groups
 May only provide superficial responses to technical questions  Workshop should last 3-5 days.
 Requires a highly trained facilitator  Must turn a group of participants into a team - this takes 1-2 days.
 Session leader makes sure each step has been completed thoroughly.
 Watch for 

Session leader steps in when there are differences of opinion - “open issues”.
Meeting room should be well-equipped for presentations, recording etc.
 sample bias
 dominance and submission
3
Knowledge Elicitation Techniques in RE The Knowledge Level Knowledge level

model
 View knowledge modelling as:
rationalise
 Background Observe behaviour of an agent as black box
 Knowledge elicitation is concerned with  It acts as if it has some knowledge about its Environment
discovering ‘expert’ knowledge environment which it uses rationally
Example
Example Techniques
Behaviour
 Grew out of Expert Systems work in the Techniques  It takes actions to achieve ascribed goals
Eliciting
Elicitingdomain
domainknowledge
knowledge observe
80’s Construct two models:
 Card Sorting
 Originally focussed on deriving expert’s  Card Sorting  Symbol Level - descriptions for mechanising Observer
 Laddering
 Laddering (modeller)
“rules” for Rule-based Systems  Proximity Scaling Techniques behaviour
 Proximity Scaling Techniques
 More recently, focussed understanding  Knowledge Level - descriptions of the agent's
mechanise
Eliciting
Elicitingperformance
performanceknowledge
knowledge
“problem solving methods” knowledge of the world
 Protocol Analysis Agent
 Protocol Analysis
Using
UsingMultiple
MultipleExperts
 But KE is hard Experts
 Delphi Technique  Two-step rationality:
 Delphi Technique
 Separation of domain knowledge from  Focus Groups Agent applies its knowledge in two stages: Symbol level
 Focus Groups
performance knowledge  Repertory Grids
 Repertory Grids  First creates a task specific model from the KL model
 Modeling problems Automated
AutomatedTechniques
Techniques model based on features of the task.
 Machine Learning
Brittleness  Machine Learning Hence, we actually need 3 models: Knowledge
Assumption of rationality
 Domain model - a systematic way of talking
 Representational Problem about a domain, with a coherent ontology. Problem
epistemological inadequacy  Task model - models goals, what it means to solving Behaviour
expressiveness vs. acquirability
achieve a goal, and how goals are related. method
 Expert Bias  Problem-solving method - a way of relating task
Task features
and domain models to accomplish goals.
Knowledge Elicitation Techniques more KE techniques

 Protocol Analysis
 based on vocalising behaviour
 Card Sorting  Laddering
 Think aloud vs. retrospective protocols  For a given set of domain  Uses a set of probes (types of
 Advantages objects, written on cards: question) to acquire structure and
 Direct verbalisation of cognitive activities  Expert sorts the cards into content of stakeholders’
 Embedded in the work context groups... knowledge.
 Good at revealing interaction problems with existing systems  ...then says what the criterion  Interview the expert.
 Disadvantages was for sorting, and what the
 Use questions to move up and
 Essentially based on introspection, hence unreliable groups were.
down a conceptual hierarchy
 No social dimension  Advantages  Advantages
 simple, amenable to automation
 Proximity Scaling Techniques  elicits classification knowledge
 deals with hierarchical knowledge,
including poly-hierarchies (e.g.,
 Given some domain objects, derive a set of dimensions for classifying them:  Problems goal trees, “is-a” taxonomies).
step 1: pairwise proximity assessment among domain elements  suitable entities need to be  knowledge is represented in
step 2: automated analysis to build multi-dimensional space to classify the objects identified with suitable semantic standardised format
 Advantages spread across domain.  can elicit structural knowledge
 help to elicit mental models, where complex multivariate data is concerned  No performance knowledge  suitable for automation.
 good for eliciting tacit knowledge  Disadvantages
 Disadvantages  assumes hierarchically arranged
 Requires an agreed on set of objects knowledge.
 Only models classification knowledge (no performance knowledge)
Source: Adapted from Hudlicka, 1996.
4
KA from Multiple Experts Abstractionism vs. Contextualism

 Delphi technique
 Abstractionism
 Used where contact between experts is difficult:
 Builds models abstracted from a domain; the model is used to answer
 Each expert submits their judgement
 All judgements are circulated anonymously to all experts questions
 Each expert then submits a revised judgement (1) Decide on the ontology of the phenomena we wish to describe
 Iterate until judgements converge (2) Use this ontology to represent the domain of discourse
 Assumes knowledge and understanding are independent from context
 Focus Groups  Used by natural scientists and engineers.
 A technique derived from marketing:  …although many scientists don’t realize that step 1 involves choice
 Assemble experts together and discuss the problem  logical positivism vs. theory-driven observation
 Discussion may be structured (e.g. debate) or unstructured
 Contextualism
 Repertory Grids (based on Kelly’s Personal Construct Theory)  Emphasizes the details and idiosyncrasies of the domain
 Used to detect terminological differences (1) Collect naturalistic data from the domain of study (Rich descriptions)
 Get the experts to agree a set of entities (2) Use the data to support explanations (but don’t build abstract models)
 Each expert provides attributes and values  Assumes it is impossible to build models that have meaning when removed
 For each attribute in expert A's grid, find the closest match in expert B's grid. from their context
(i.e. are there attributes which have the same discriminatory function?)  Used by many social scientists
 Experts then rate the entities using each other's attributes  but generally limits them to the descriptive rather than predictive/prescriptive
Participant Observation Ethnomethodology

 Basis
 Approach  Social world is ordered
 Observer spends time with the subjects  The social order may not be obvious, nor describable from common sense
 Joining in long enough to become a member of the group  The social order cannot be assumed to have an a priori structure
 Hence appropriate for longitudinal studies  Social order is accomplished on a moment-to-moment basis through participants’
collective actions (rather than through any pre-existing structures)
 Advantages  i.e. social order only observable when an observer immerses herself in it.
 Observation should be done in a natural setting
 Contextualized;
 Need to consider how meanings develop and evolve within context
 Reveals details that other methods cannot
 “Use the members’ own Categories”
 Disadvantages  Most conventional approaches assume preexisting categories
 Extremely time consuming!  This may mislead the observer (e.g. appropriation)
 Resulting ‘rich picture’ is hard to analyze  Ethnomethodology attempts to use the subjects’ own categories
 Cannot say much about the results of proposed changes  What categories (concepts) do they use themselves to order the social world?
 What methods do people use to make sense of the world around them?
 Watch for  During observation, use the same methods members use, eg by developing a
legitimate role within the community under observation.
 going native!
 Measurement
 No scientific objectivity, so use the subjects’ own measurement theory
5
Ethnomethodological approach Postscript: Postmodernism

 Ethnomethodology is a subarea of Anthropology  Modernism  Postmodernism (PoMo)
 Looks for behaviours that may be different in a specific culture but which  Rationality is the highest form of  Questioning the grand narrative
have the same underlying purpose or meaning. mental functioning  A grand narrative is a story that a
E.g. how do people go about gaining status in different cultures: culture/society tells itself about it’s
  Modern science produces universal
practices and beliefs
 Frenchmen brag about sexual conquests to gain status; truths
 E.g. in the US: “democracy is the most
 Americans brag about money to gain status.  …independent from the context and enlightened/rational form of government”
 Each of these topics is taboo in the other culture status of the scientist who produced
 E.g. in science: “scientific truths are
them
universal and eternal”
 Uses a very tightly controlled set of methods:  Rationality will always lead to progress  Postmodernism identifies and critiques
and perfection such narratives
 Conversational analysis
 All human institutions can be scientifically  Instead, look for mini-narratives
 Measurement of body system functions - e.g. heartbeat analyzed and improved  Stories that explain small practices, local
 Studies of Non-verbal behaviour (e.g. gestures, body language)  Reason is the ultimate judge of what is events, situated, contingent behaviour
 Detailed video analysis right (true, legal, ethical,…)  …and don’t make any claims about
 These techniques are useful in capturing information about a social setting. universality, truth, or stability
 Language must be rational
 E.g. Literary Deconstruction
Other observation techniques can be applied:
 it only exists to represent the real world;
  there must be a firm, objective  Examine what a text does not say, what
it represses
 Time-motion study connection between the “signifier” and
the “signified”  Reveal internal arbitrary hierarchies and
 who is where, when? dichotomies
 the meaning cannot depend on the
 Communication audit audience  E.g. Semiotics
 who talks to whom about what?  The study of the relationship between
 Use of tools - status symbols plus sharing rules signs and the things they signify
Source: Adapted from http://www.colorado.edu/English/ENGL2012Klages/pomo.html
University of Toronto Department of Computer Science
What has PoMo got to do with RE?

 logical positivist view:
 “there is an objective world that can be modeled by building a consistent body of
knowledge grounded in empirical observation”
 In RE: “there is an objective problem that exists in the world”
 Build a consistent model; make sufficient empirical observations to check validity
 Use tools that test consistency and completeness of the model
 Use reviews, prototyping, etc to demonstrate the model is “valid”
 Popper’s modification to logical positivism:

 “theories can’t be proven correct, they can only be refuted by finding
exceptions”
 In RE: “requirements models must be refutable”
 Look for evidence that the model is wrong
 E.g. collect scenarios and check the model supports them
 post-modern view:
 “there is no privileged viewpoint; all observation is value-laden; scientific
investigation is culturally embedded”
 In RE: “validation is always subjective and contextualised”
 Use stakeholder involvement so that they ‘own’ the requirements models
 Use ethnographic techniques to understand the participant’s worldviews
© 2000-2004, Steve Easterbrook 23

Lecture 4: Requirements Elicitation Elicitation Techniques: University of Toronto University of Toronto

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lecture 4: Requirements Elicitation Elicitation Techniques: University of Toronto University of Toronto

Uploaded by

Copyright:

Available Formats

University of Toronto Department of Computer Science University of Toronto Department of Computer Science

Lecture 4: Requirements Elicitation Elicitation Techniques

Background Reading “Hard Data” Collection

Interviewing Tips Surveys and Questionnaires

 Ask if you can record the interview  Disadvantages

 Ask easy questions first  Watch for:

Survey Sources of Error Meetings

© 2000-2004, Steve Easterbrook 9 © 2000-2004, Steve Easterbrook 10

Group Elicitation Techniques Joint/Rapid Application Development

Knowledge Elicitation Techniques in RE The Knowledge Level Knowledge level

Knowledge Elicitation Techniques more KE techniques

KA from Multiple Experts Abstractionism vs. Contextualism

© 2000-2004, Steve Easterbrook 17 © 2000-2004, Steve Easterbrook 18

Participant Observation Ethnomethodology

Ethnomethodological approach Postscript: Postmodernism

University of Toronto Department of Computer Science

What has PoMo got to do with RE?

 Popper’s modification to logical positivism:

You might also like