Chapter Three

Research Design: Planning the

research
Major topics
 The three research designs (approaches) and comparison
 Sampling and related issues
 Details of Qualitative research approaches
 Details of Quantitative research approach
 Details of Design science research approach
 Planning the Research Project
 A general strategy = research design (RD)
 RD provides /includes
 Philosophical assumption, Mainly Research approach/ and specific
method
 Overall structure for the procedures
 The data the researcher collects/and techniques
 The data analyses/experimentation/design the researcher conducts
 Evaluation…
 Must plan the overall design carefully to be successful
 Wasted effort if half-prepared (vague ideas)
 Must identify RESOURCES, PROCEDURES, and DATA always with the
goal from the very beginning
 It matters to get your proposal accepted or rejected
Cont…
A research approach/method is a strategy of inquiry which
moves from the underlying philosophical assumptions to
specific research design and data collection & analysis
 Research methodologies reflect high-level approaches to
conducting research.
 The individual steps within the methodology might vary
based on the research problem being performed.
 There are three broad approaches to be used in research design
 Quantitative.
 Qualitative.
 Design science

4
Philosophical Assumption: Positivist Vs
Interpretive
 Positivists generally assume that reality is objectively given
and can be described by measurable properties which are
independent of the observer (researcher) and his or her
instruments.

 Interpretive researchers start out with the assumption that

access to reality (given or socially constructed) is only through
social constructions such as language, consciousness and
shared meanings.
Research Approach: Qualitative
 Qualitative research involves the use of qualitative data, such
as interviews, documents, and participant observation data, to
understand and explain social phenomena.

 Examples of qualitative methods are action research, case

study research and ethnography.

 Qualitative data sources include observation and participant

observation (fieldwork), interviews and documents and texts,
and the researcher's impressions and reactions (Myers 2009).
Research Approach: Quantitative
 Quantitative research methods involves the use of
quantitative data (numerically represented) and were originally
developed in the natural sciences to study natural phenomena.

 Examples of quantitative methods now well accepted in the

social sciences include survey methods, laboratory
experiments, formal methods (e.g. econometrics) and
numerical methods such as mathematical modeling

 The most popular data collection techniques include:

Questionnaire (survey instruments) , secondary data sources or
archival data, objective measures or tests.
Research Approach: Design
Science(engineering research)
 Is a problem-solving paradigm:

 Seeks to create innovations that define the ideas, practices,

technical capabilities, and products through which the
analysis, design, implementation, and use of IT can be
effectively and efficiently accomplished.

 The specfic method can be named as “ design and evaluate“

 Formulative research/design oriented

 Employes data collection techniques from qualiatative and

quantitative methods
It is recommended to apply
Triangulation
Combining Methods, Data
Some comparisons of the three research
approaches
Quantitative Qualitative Design science
•Explanation, •Explanation, •Motivated to solve
prediction description problems
•Test/Modify Build theories Prescriptive – Develop
theories artifact
•Known variables Unknown variables Known variables
•Large sample Small sample Small sample, case
based
•Standardized Observations, Observation,
instruments interviews interviews, etc
•Deductive Inductive Inductive

10
 Cont’d
Quantitative Qualitative Design science
Statistical •Analytic •Replicable to similar
generalization generalization problems in similar
contexts
•Use numbers Use texts Texts, codes and
drawings
•Standard No standard Pragmatic validity
instrument to instrument,
validation triangulation
•Objective Truth Subjective truth Subjective Truth ???
•Linear Process Iterative Iterative
12
Sampling – Core decision in research design
 Do we need sample first of all?
 Define Population – potential data sources
 Data Sources (-Sampling )
 People (respondents)
 Databases, Computer Log files
 Documents (web documents, news…, reports …)
 Devices like camera
 All require a decision whether to use as is or sample.
 Why sample? How? How many?
 Sampling and Data collection techniques should agree and
 Depends on the type of research methods and type of data you
want to collect
 What is Sampling?
• The act, process, or technique of selecting a
suitable sample, or a representative part of a
population for the purpose of determining
parameters or characteristics of the whole
population.
• There would be no need for statistical theory if a
census rather than a sample was always used to
obtain information about populations.
• A census may not be practical and is almost
never economical.
Sampling
 Sampling Techniques
• Probability sampling
– Each unit of the population will be
represented in the sample.
– Each member of the population has a chance
(equal in the case of random selection) of
being selected.
• Non-probability sampling
– The researcher has no way of forecasting that
each member of the population will be
represented in the sample.
– Some members of the population have little or
no chance of being selected.
 Sampling Techniques … cont’d

Probability Sampling
– Simple random sampling
– Systematic random sampling
– Stratified random sampling
– Proportional stratified sampling
– Cluster sampling
 Probability Sampling Techniques … cont’d

• Simple random sampling

– The least sophisticated of all sampling designs
– Simple random selection where every member
of the population is given an equal chance of
being selected;
– Good for homogeneous population;
– Easy when the population is small and
elements are known;
– Impractical for very large populations
 Probability Sampling Techniques… cont’d

• Systematic random sampling

– Selecting elements of the population in
predetermined sequence;
– Select every kth item on a list (k= N/n)
– Randomness element is in picking up the starting
point
 Probability Sampling Techniques… cont’d
• Stratified random sampling
– Applied when the population has different
layers (strata)
– The researcher samples from each one of the
layers (stratum) equally
– Examples
• Sampling of school children from grades 4, 5 and 6
• Sampling of customers
– Corporate customers
– Residential customers
– etc
Probability Sampling Techniques … cont’d

• Proportional stratified sampling

– When number of the elements of the strata
are different
• 2,000 corporate customers
• 400,000 residential customers
Probability Sampling Techniques … cont’d
• Cluster sampling
– Grouping the population into clusters and then
select members of clusters
Random selection
Population of clusters
C1 C2 C3 C4 C5

C5 C6 C7 C8 C8
C9 C10 C11 C12 C11
 Probability Sampling Techniques …cont’d
Population
characteristics
Homogeneous members
Stratified population with
approximately equal in size
Stratified population, strata
different in size
Population with discrete
clusters with similar
characteristics
Appropriate sampling
technique
•Simple random sampling
•Systematic random
sampling
Stratified random
sampling
Proportional stratified
sampling
Cluster sampling
 Non-Probability Sampling Techniques

• Non-probability sampling
– Convenience sampling
– Quota sampling
– Purposive sampling
Non-Probability Sampling Techniques… cont’d

• Convenience sampling
– Taking of samples that are readily available
• Ex: those that arrive on a scene by coincidence
– May be appropriate for some less demanding
research
• Purposive sampling
– Units of the sample are chosen purposively.
– Choosing people who we have decided are
“typical” of a group;
 Non-Probability Sampling Techniques… cont’d
• Quota sampling
– A variation of convenience sampling
– Elements are selected in the same proportion as in
the population but not in a random fashion
• Ex: there are equal number of Information
Science (IS) and Computer Science (CS) students
–Quota sampling would choose 20 IS and 20 CS
students without any attempt to random
selection
What should be Sample Size?

How Many?

Which?
Key features: Sufficient and
Representative
Identifying a Sufficient Sample Size
 It depends on many factors like type of research …..
 To some extent, the size of an adequate sample depends on how homogeneous or
heterogeneous the population is – how alike or different its members are with respect
to the characteristics of research interest
 Important, too, is the degree of precision with which the researcher wishes to
draw conclusions or make predictions about the population under study
 The basic rule is, the larger the sample, the better
 But such a generalized rule is not too helpful to a researcher
who has a practical decision to make about a specific research
situation
 Statisticians have developed formulas for determining the
desired sample size for a given population

 There are also some guidelines like shown in the net slide
 Cont…
 The following are guidelines for selecting a sample size for
quantitative survey:
 For small populations (with fewer than 100 people or other units), there is little
point in sampling. Survey the entire population
 If the population size is around 500, 50% of the population should be sampled
 If the population size is around 1,500, 20% should be sampled
 Beyond a certain point (at about 5,000 units or more), the population size is
almost irrelevant, and a sample size of 400 should be adequate
 Generally speaking, then, the larger the population, the smaller the percentage
(but not the smaller the number!) one needs to get a representative sample
 For quantitative laboratory experimental -even more
 In DM like - ~`5000 records
 Does this work for qualitative? N0
 Most researches in qualitative approaches uses 1 – 30 sample size.
 Actually the concept of saturation determines sample size in qualitative
researches.
 points to recall
• Research Design: Planning the research
• Qualitative
• Case study and action research using interview, observation,
document analysis
• Quantitative
• Laboratory experiment and survey using data from database or
collected through questionnaire
• Design Science
• create innovations that define the ideas, practices, technical
capabilities, and products using qualiatative or quantitative data
• Triangultion
• Coparisons of the three resercah approaches
• Data sources and sampling
Qualitative Research Design
Qualitative Research Methods
 Its aim is to give a complete, detailed descriptions of
the phenomena to be studied
• Objective facts + values
 Key concern and philosophical assumption
• understanding how people make sense of their
worlds and the experiences people have
• knowing or understanding from the participants’
perspectives
 Key focus –
• understanding (rather than predicting or controlling)
social settings or social phenomena
When to choose qualitative approach?
 Describe the phenomena
 In better detail
 Identifying inputs/ requirements based on empirical base
 Build a theory
 To gain new insights about a particular phenomena
 Develop new concepts or theoretical perspectives about the
phenomena
 Discover the problem that exists in the phenomena
 Verification – to check the validity of certain assumptions (not
hypothesis), claims, generalization with the real world
 Evaluation – to evaluate the effectiveness of a particular
policies, design artifacts, programs, etc
 Define problem

Review literature Conceptual

framework/
Proposal
Collect Data

Data Analysis Build

understanding
/ Theory
Data Interpretation/ Or Framework
Report Findings
Common Types of Qualitative Research
Designs
 There are many qualitative research methods/designs.
 Case study
 Action research
 Ethnography
 Etc.
Case study research method/design
 Its main purpose is to gain a deep understanding of a phenomenon
 Example:
 To understand the capability of a new tool
 To identify factors affecting communication in code
inspections
 To characterize the process of coming up to speed on a project
 In building a framework, model or any design
 Objective of Investigation
 Exploration-To find what’s out there
 Characterization-To more fully describe
 Validation-To find out whether a solution/ proposition is true
Cont…
 Subject of Investigation in a case study could be
 An intervention, e.g. tool, technique, method, approach to
design, implementation, or organizational structure

 An existing thing or process, e.g. software implementation,

project success, defects, network performance …
Important aspects of a Case Study
Research design/process
 Five main parts of a case study research design (under the
umbrella of the general research process)
 Research questions
 Propositions (if any)
 Unit(s) of analysis
 Logic linking the data to the propositions (analysis)
 Criteria for interpreting the findings
Study Questions(RQ)
 Case studies are most appropriate for research questions that
are of the “how” and “why” variety
 The initial task is to clarify precisely the nature of the study
questions (i.e. make sure they are actually “how” or “why”
questions)
 Examples:
 "Why do developers prefer this tool/model/notation?"
 "How are software inspections carried out in practice?“
 "How does agile development work in practice?"
 "Why do programmers fail to document their code?“
 "How does software evolve over time?“
 "Why have formal methods not been adopted widely for
safety critical applications?“
 "How does a company identify which software development
projects to start?"
Study Propositions (hypothesis)
 Propositions are common than hypothesis in qualitative studies
 Propositions are statements that help direct attention to something
that should be examined in the case study, i.e. point to what should
be studied
 Example: “Organizations collaborate because they derive mutual
benefits”
 Propositions will tell you where to look for relevant evidence
 Example: Define and ascertain the specific benefits to each
organization
 Some studies may not have propositions –this implies a topic of
“exploration”
 Note: Even exploratory studies should have both clearly-stated
purposes and clearly-stated criteria for success
Unit of Analysis
 The unit of analysis defines what a “case” is in a case study
 Example: a unit of analysis (case) may be an individual, and
the case study may be the life history of that person
 Other units of analysis include decisions, social programs,
processes, changes, methods…
 Note: It is important to clarify the definition of these cases as
they may be subjective, e.g. the beginning and end points of
a process
 What unit of analysis to use generally depends on the primary
research questions
 Once defined, the unit of analysis can still be changed if
desired, e.g. as a result of discoveries based on data
 To compare results with previous studies (or allow others to
compare results with yours), try to select a unit of analysis that
is or can be used by others
 Examples of Units of Analysis
 For a study of software adoption
 Individuals
 Development team
 Organization
 For a study of programming
 Programming episode
 Pairs of programmers
 Development team
 Organization
 For a study of software evolution
 Modification report
 File
 System
 Release
 Stable release
Linking Logic (analysis)
 Logic or reasoning to link data to propositions/and/or findings
 One of the least well developed components in case studies
 Many ways to perform this, but none as precisely defined as
the treatment/subject approach used in experiments/ survey
 Can be simply explained as interrelating themes. Based on
either open or closed codes
 One possibility is pattern matching: describe several potential
patterns, then compare the case study data to the patterns and
see which one is closer
Cont…
 Criteria for High Quality Analysis
 Present all the evidence
 Develop rival hypotheses/concepts
 Address all major rival interpretations
 Address most significant aspect of the case study
 Use prior or expert knowledge
Criteria for interpreting the findings
 The last issue in case study design is
 Setting criteria for interpreting findings
 Obviously from the context /case at hand
 And with the existing literature.

 It is nothing but planning how the findings should be

interpreted
 Action research design/process
• Action research is either research initiated to solve an immediate
problem or a reflective process of progressive problem solving led
by individuals working with others in teams or as part of a
"community of practice" to improve the way they address issues
and solve problems.

• is a disciplined process of inquiry conducted by and for those

taking the action. The primary reason for engaging in action
research is to assist the “actor” in improving and/or refining his or
her actions.

• Action research involves actively participating in a change

situation, often via an existing organization, whilst simultaneously
conducting research
Cont…
 "Action research, also known as participatory research, is learning
by doing in the sense that a group of workers identify a problem,
develop a resolution, implement the solution and then analyze the final
results.
 The action research process consists of five steps in the cycle:
diagnosing, action planning, taking action, evaluating, and specifying
learning. 
 Action research differs from conventional research methods in three
fundamental ways.
 First, its primary goal is social change.
 Second, members of the study sample accept responsibility for
helping resolve issues that are the focus of the inquiry.
 Third, relationships between researcher and study participants are
more complex and less hierarchical
Qualitative research in IT/CS
 Used to understand user problems empirically for design such
as in diagnosing user problems and needs
 Used to collect inputs in scientific manner for artifact
development
 Used in artifact evaluation- researchers qualitatively evaluate a
product by interviewing and observation
 Used to uncover non-technical factors affecting the adoption
and evolution of a new software product and other IT systems
 Used to develop theories such as HCI theory
Application of Qualitative - Example
 System Development Research Process that Nunamaker, et al
(1991) proposed five stages or activities
1. construct a conceptual framework,
2. develop a system architecture,
3. analyze and design the system,
4. build the (prototype) system, and
5. observe and evaluate the system.
 The last stage (along with the first) explicitly includes
“Develop new theories/models based on the observation and
experimentation of the system’s usage”
Points to recall
 Research Design can be quantitative, qualitative, design based or a
combination of these
 The selection Should be logical for the validity and reliability of
the research.
 There are different methods under each research approach
 The choice will mainly depend on the research agenda/topic
 Determining data sources and type of data to be collected is
commonly important to all
 Deciding if sampling is required and selecting appropriate sampling
method with right number of sample is again important in all cases.
 Case study and action researches are good examples of methods
under Qualitative approach
Quantitative Research Design
Quantitative Research Methods
 Uses deductive method of knowledge acquisition
 Intends to falsify an existing theory
 Tries to generate statistically generalizable knowledge
 Accepts objectivity of knowledge
 Uses a standard measurement and validation instruments
 In both laboratory experiment and survey
 Quantifies the phenomena in terms of numbers
 The researcher is independent in the process of knowledge construction
 Quantitative methods emphasize objective measurements and
the statistical, mathematical, or numerical analysis of data collected
through polls and questionnaires, or by manipulating pre-existing
statistical data using computational techniques. 
Cont…
 Your goal in conducting quantitative research study is to determine
the relationship between one thing [an independent variable] and
another [a dependent or outcome variable] within a population.
 Quantitative research designs are two types
 Descriptive [subjects usually measured once]
 A descriptive study establishes only associations between
variables;
 Experimental [subjects measured before and after a treatment]
 Survey experiment or laboratory experiment
 An experimental study establishes causality.
Cont…
 Quantitative research may focus in one or another of the
following
 Descriptive survey (fact finding)
 Explorative/Experimental survey (looking for patterns)
 Analytical (explaining why or how)
 Predictive (forecasting the likelihood of particular events)
Descriptive survey
 It started during the Roman Empire
 Took census to know the number of people under its empire
 Used for taxation purposes
 Seeks to accurately describe current or past phenomena - to
answer such questions as:
a) What type of open source software frequently used in
Ethiopia?
b) What type applications usually consume more bandwidth?
c) What type of computers mostly imported in Ethiopia
d) What are the main software tools used by end users?
Experimental survey /Explorative
Seeks to understand psychological and sociological phenomena in
an effort to find behavioral patterns:
a) How do users use the new software system?
b) Why would users use the software in this way contrary to
original purpose. For example use of Internet for news
rather work related matters
c) What are the antecedent factors for IT project success?
Analytical
 Seeking to explain the reasons behind a particular occurrence by
discovering causal relationships.
 Once causal relationships have been discovered, the search then
shifts to factors that can be changed (variables) in order to influence
the chain of causality. Typical questions are:
a) Do user involvement in software development and users IT skills increase
software success?
b) Does Hardware performance have an influence on software failure?
c) Are road related factors more important in determining accident severity?
d) Can network signal be improved by ….
e) Does hybrid approach provide better performance.?
Predictive
 Seeks to forecast the likelihood of particular phenomena
occurring in given circumstances.
a) Does internet bandwidth predict growth online business services ?
b) Can the new software system increases the organization’s efficiency?
c) To what extent does new software predict growth of the organization
profit?
d) How to predict network attacks
 It compares independent (IV) and dependent variable (DV).
 We then predict DV by measuring IV
 Examples of Quantitative methods
 Two most widely used Methods are
 Laboratory Experiment
 Survey
 Experimental study (laboratory experiment)
 Mostly data is from database
 Systematic and appropriate data access has to be maintained
 Analysis
 Different parameter settings and various experiments
 Main question-How many experiments ?

 Using machine learning algorithms (for NLP, DM, IR….)

 Using network analysis tool (simulators, emulators, test beds..
 Validity through standard testing
Cont…
 It is used in IS/CS in several deferent fields: BI, DM, artificial
neural net-works, automating theorem proving, natural
languages, analyzing performances and behaviors, etc.
 It is important to state that all the experiments and results
should be reproducible.
 Concerning, for example, network environments with several
connection re-sources and users, experiments are important
methodology (complexity)
 Also in IS/CS fields that take in consideration the human
factor (Human-Computer Interaction, for example), it is
mandatory the usage of experimental approaches.
Cont…
 Survey Study
 Data through questionnaires (standard instruments)
 from human data sources
 Analysis -Statistical - two types
 Descriptive statistics
 Used to describe the phenomena with frequency, mean,
median, mode etc…
 Inferential statistics
 Is used to infer about the population from the sample data and
show variable relationship
 T-test to compare means of two groups
 Analysis of Variance (ANOVA)- (When groups more than
two), Regression , ……
 Validity through standardization of instruments and analysis
Enumerate basic differences of qualitative
research design and quantitative research
design?
Design (based/science) Research
Topics
 Conceptualizing design and design research
 Research process in design perspective
 Design research output
 Design research guidelines
 Knowledge contribution of the design research
Conceptualizing design and design
research
 Sciences of the Artificial, 3rd Ed. – Simon 1996
 A Problem Solving Paradigm
 The Creation of Innovative Artifacts to Solve Real Problems
 Design in Other Fields – Long Histories
 Engineering, Architecture, Art
 Role of Creativity in Design
 DSR in IT/Information Systems
 A. Hevner, S. March, J. Park, and S. Ram, “Design Science Research
in Information Systems,” Management Information Systems Quarterly,
Vol. 28, No. 1, March 2004, pp. 75-105.
 S. Gregor and D. Jones, “The Anatomy of a Design Theory,” Journal
of the Association of Information Systems, (8:5), 2007, pp. 312-335.

68
Cont…
 Information Systems (IS) , CS and IT works are complex,
artificial, and purposefully designed.
 IS are composed of people, structures, technologies (HW, SW,
Network)and work systems.
 Two Basic IT Research Paradigms
 Behavioral Research – Goal is Truth
 Design Research – Goal is Utility

69
Cont…
 Design
 Design is an Artifact (Noun)
 Constructs
 Models
 Methods
 Instantiations
 Design is a Process (Verb)
 Build
 Evaluate
 Design is a Wicked Problem
 Unstable Requirements and Constraints
 Complex Interactions among Subcomponents of Problem and
resulting Subcomponents of Solution
 Inherent Flexibility to Change Artifacts and Processes
 Dependence on Human Cognitive Abilities - Creativity
 Dependence on Human Social Abilities - Teamwork
70
Cont...
 Design research basics
 Process model
 Artifact types:
 result of the research work
 Design research perspectives
 Artifact structure
 content of the research approach
 Evaluation:
 evaluation criteria and evaluation approach
Process model
 Design research a problem-solving paradigm:
 seeks to create innovations that define the ideas, practices,
technical capabilities, and products through which the
analysis, design, implementation, and use of IT or
information systems can be effectively and efficiently
accomplished [Tsichritzis 1997; Denning 1997]

 Diffrent preocess representations are proposed in litartures

 Peffer etal (2008)
 Takeda (1990)
 Gregor and Hevner (2013)
Peffer etal (2008) In action
 Design research process

knowledge
flows + operation and goal knowledge

circumscription
process
steps
Awareness of
Suggestion Development Evaluation Conclusion
problem

logical
formalism
abduction deduction

[Takeda,1990]
 In this model all design begins with Awareness of a problem.

Design research is sometimes called “Improvement Research” and

this designation emphasizes the problem- solving/performance-
improving nature of the activity.

Suggestions for a problem solution are abductively drawn from

the existing knowledge/theory base for the problem area (Pierce,
1931).

An attempt at implementing an artifact according to the suggested

solution is performed next.
This stage is shown as Development in the diagram.
Partially or fully successful implementations are then Evaluated
(according to the functional specification implicit or explicit in the
suggestion).

Development, Evaluation and further Suggestion are frequently

iteratively performed in the course of the research (design) effort.

The basis of the iteration, the flow from partial completion of the
cycle back to Awareness of the Problem, is indicated by the
Circumscription arrow.

Conclusion indicates termination of a specific design project.

New knowledge production is indicated in Figure by the arrows
labeled Circumscription and Operation and Goal Knowledge. 

The Circumscription process is especially important in understanding

design research because it generates understanding that could only
be gained from the specific act of construction.

Circumscription is a formal logical method (McCarthy, 1980) that

assumes that every fragment of knowledge is valid only in certain
situations.
Further, the applicability of knowledge can only be determined
through the detection and analysis of contradictions – in common
language, the design researcher learns or discovers when things
don’t work “according to theory."

This happens many times not due to a misunderstanding of the

theory, but due to the necessarily incomplete nature of ANY
knowledge base.

The design process, when interrupted and forced back to

Awareness of Problem in this way, contributes valuable
constraint knowledge to the understanding of the always-
incomplete-theories that abductively motivated the original
design.  
Another Perspective – to wards the DSR Process
Useful Knowledge in general
(represented as omega and lambda)
Ω – Descriptive Λ – Prescriptive
Knowledge Knowledge

• Phenomena (Natural,
Artificial, Human)
• Observations
• Classification
• Measurement
• Cataloging
• Sense-making
• Natural Laws
• Regularities
• Principles
• Patterns
• Theories

79
The DSR Process

Application Human
Ω Knowledge Capabilities
Environment

Knowledge - Cognitive
Sources - Creativity
- Reasoning
- Research - Analysis
Opportunities Contribution to Informing - Synthesis
and Problems Ω Knowledge Λ Knowledge
- Social
- Research - Teamwork
Questions - Collective
Intelligence

Constructs Models Methods Instantiations

Design Theory

Λ Knowledge

80
Knowledge Growth in DSR Cycles
Design Cycle 1 Design Cycle 2 Design Cycle n
Ω1 Knowledge Ω2 Knowledge Ωn Knowledge

…
…

…
…

Λ1 Knowledge Λ2 Knowledge Λn Knowledge

81
 Three Cycles of DSR

Environment Design Science Knowledge Base

Application Domain Foundations

Build Design • Scientific Theories
• People
Artifacts & & Methods
•Organizational Processes
• Experience
Systems
Relevance Cycle Rigor Cycle & Expertise
• Technical Design
Systems • Requirements Cycle
• Grounding

• Field Testing
• Additions to KB
•Problems
& Opportunities Evaluate •Meta-Artifacts
(Design Products &
Design Processes)

82
 Cont…
 The Relevance Cycle
 The Application Domain initiates Design Research with:
 Research requirements (e.g., opportunity, problem, potentiality)
 Acceptance criteria for evaluation of design artifact in application
domain
 Field Testing of Research Results
 Does the design artifact improve the environment?
 How is the improvement measured?
 Field testing methods might include Action Research or Controlled
Experiments in actual environments.
 Iterate Relevance Cycle as needed
 Artifact has deficiencies in behaviors or qualities
 Restatement of research requirements
 Feedback into research from field testing evaluation

83
Cont…
 The Rigor Cycle
 Design Research Knowledge Base
 Design Theories
 Engineering Methods
 Experiences and Expertise
 Existing Design Artifacts and Processes
 Research Rigor is predicated on the researcher’s skilled
selection and application of appropriate theories and methods
for constructing and evaluating the artifact.
 Additions to the Knowledge Base:
 Extensions to theories and methods
 New experiences and expertise
 New artifacts and design processes

84
Cont…
 Design Cycle
 Rapid iteration of Build and Evaluate activities
 The hard work of design research (1% inspiration and
99% perspiration - Edison)
 Build – Create and Refine artifact design as both product (noun)
and process (verb)
 Evaluation – Rigorous, scientific study of artifact in laboratory or
controlled environment
 Continue Design Cycle until:
 Artifact ready for field test in Application Environment
 New knowledge appropriate for inclusion in Knowledge
Base

85
Cont...
 Artifacts
 What are some of the examples of design research
output??
Design research outputs
[March & Smith, 1995]
 Constructs
 conceptual vocabulary of a problem/solution domain
 Methods
 algorithms and practices to perform a specific task
ut
 Models
t p
u
 a set of propositions or statements expressing relationships among
o
constructs
si s
 abstractions and representations
he
 Instantiations T
 constitute the realization of constructs, models and methods in a
working system
 implemented and prototype systems
 Better theories
 artifact construction
Cont…
 Specific Examples
 Methods
 Derive superior algorithms ( in NW, IR, Machine learning..
 Develop new methodology (for software develop..)
 Models
 Provide a unifying framework
 Developing better architecture
 Instantiations
 Produce an ambitious system
 Develop a new tool (DM/BI, protocol, KBS, Stemmer…

The major requirement of Computer Network Research and

development process is to design a new protocol or algorithm and in
further steps to validate and verify their correctness of action and
their deployment feasibility (Kushwaha, 2014)
Design research outputs
constructs
better theories
emergent theory about models
embedded phenomena
abstraction models
abstraction methods
knowledge as
constructs
operational principles
better theories

abstraction

artifact as situated implementation instatiations

methods
constructs
[Purao , 2002]
Gregor and Hevner (2013)
Design Theory as Knowledge
 Theory – set of statements of relationships among constructs that
aims to describe, explain, enhance understanding, and, in some
cases, allow predictions about the future (Gregor 2006)
 Design Theory (Gregor and Jones 2007) – prescriptions for
design and action
 Behaviors of individual artifacts (Level 1) lead to empirical
generalizations or technical rules
 Extending the boundaries of abstract design artifacts (Level 2) grows
nascent (promising) design theory
 Middle-range design theory (Level 3) – understanding expands to partial
theory (Goal of theory in applied fields – Merton 1968)
 Grand design theory – all encompassing truths

91
Environment Relevance IS/CS Research Rigor Knowledge Base

People Foundations
•Roles Develop / Build •Theories
•Capabilities •Theories •Frameworks
•Characteristics •Artifacts •Experimental
Instruments
•Experience
Business Applicable •Constructs
Organizations •Models
•Strategies Needs Knowledge •Methods
•Structure Assess Refine •Instantiations
•Culture Methodologies
•Processes •Experimentation
Technology Justify / Evaluate •Data Analysis
•Infrastructure •Analytical Techniques
•Case Study •Formalisms
•Applications
•Experimental •Measures
•Communications •Validation Criteria
Architecture •Field Study
•Optimization
•Development •Simulation
Capabilities

Application in the Additions to the Knowledge

Appropriate Environment Base

92
Design Research Guidelines
Guideline Description

Guideline 1: Design as an Artifact Design-science research must produce a viable

artifact in the form of a construct, a model, a
method, or an instantiation.

Guideline 2: Problem Relevance The objective of design-science research is to

develop technology-based solutions to
important and relevant business problems.

Guideline 3: Design Evaluation The utility, quality, and efficacy of a design artifact
must be rigorously demonstrated via well-
executed evaluation methods.

Guideline 4: Research Contributions Effective design-science research must provide

clear and verifiable contributions in the areas
of the design artifact, design foundations,
and/or design methodologies.

Guideline 5: Research Rigor Design-science research relies upon the

application of rigorous methods in both the
construction and evaluation of the design
artifact.

Guideline 6: Design as a Search The search for an effective artifact requires utilizing
available means to reach desired ends while
Process satisfying laws in the problem environment.

Guideline 7: Communication of Design-science research must be presented

effectively both to technology-oriented as well
93 Research as management-oriented audiences.
Tuesday class
DSR Knowledge Contribution
Framework
 A framework for Positioning DSR with respect to Knowledge
Contribution
 Two dimensions:
 Maturity of Application Domain (Opportunities/ Problems)
 Maturity of Solutions (Existing Artifacts)
 Difficulties:
 Subjectivity – where to draw the lines
 Everything builds on something else, nothing entirely new

94
 Improvement: Develop Invention: Invent new

Low
new solutions for known
problems
Research Opportunity
Solution Maturity
solutions for new
problems
Research Opportunity

Routine Design: Apply Exaptation: Extend known

solutions to new problems
High

known solutions to known

problems (e.g. Adopt solutions from
other fields)
Research Opportunity

High Low
Application Domain Maturity
95
Invention Quadrant
 An invention is a radical breakthrough; a departure from accepted
ways of thinking and doing
 DSR projects in which little understanding of the problem context
exists and no effective artifacts are available as solutions
 Research contributions are novel artifacts or inventions
 Top level artifacts
 The newness of artifact makes this research difficult to publish
 Insufficiently grounded in theory
 Design is incomplete and not fully evaluated
 Understanding is insufficient to provide new contribution to theory via
the design

96
Invention Exemplars
 Agrawal, R., Imielinski, T. and Swami, A. (1993). “Mining Association
Rules between Sets of Items in Large Databases”, Proceedings of the
1993 ACM SIGMOD Conference, Washington DC, May.
 Aim: produce an algorithm that generates all significant association rules between
items in the database
 Practical importance: Allows organizations to find interesting relationships (e.g.
shopping patterns)
 Theoretical significance (newness): Shows (Sect 5) that no other work has done same
thing
 Description of new method: Shows requirements (Sect 1), new concepts (association
rule, support, confidence), Formal Model (pseudocode) (Sects 2-3)
 Proof: Experiments (Sect 4)
 Scott-Morton (1967) – Decision Support Systems

97
Improvement Quadrant
 An improvement is a better artifact solution in the form of
more efficient and effective products, processes, services,
technologies, or ideas
 DSR projects in which the problem context is mature but
there is a great need for more effective artifacts as solutions
 Improvement DSR is judged by:
 Clearly grounding, representing, and communicating the new
artifact design
 Convincing evaluation providing evidence of improvements over
current solutions
 All levels of artifact knowledge contribution can be made

98
Improvement Exemplars
 Many DSR in IT are in the Improvement Quadrant, for
example:
 Better data mining algorithms for knowledge discovery (extending
the initial ideas invented by Agrawal et al. (1993)); for example,
(Fayyad et al. 1996; Zhang et al. 2004; Witten et al. 2011)
 Improved recommendation systems for use in e-commerce; for
example (Herlocker et al. 2004; Adomavicius and Tuzhilin 2005)
 Better technologies and use strategies for saving energy in IT
applications; for example (Donnellan et al. 2011; Watson and
Boudreau 2011)
 Improved routing algorithms for business supply chains; for
example (van der Aalst and Hee 2004; Liu et al. 2005)

99
Improvement Exemplar - Iversen et al. 2004
 Iversen, J., L. Mathiassen, and P. Nielsen (2004) “Managing Process Risk in
Software Process Improvement: An Action Research Approach”, MIS Quarterly,
(28)3, pp. 395-434.
 Introduction
 Aim – develop a risk management approach in s/w process improvement (SPI)
 Literature Review
 Reviews literature on s/w process improvement, s/w risk management (known
problems) including existing artifacts
 Conclude – currently no comprehensive approach for managing risk in SPI
 Methodology
 Action research (described at length) Research process
 Describes 4 iterations
 Artifact description (termed research results)
 Shows strategies for managing risks in SPI teams
 Discussion
 Discusses action research process
 Claims contribution to theory – advancement of state-of-the-art in SPI

100
Exaptation Quadrant
 An exaptation is the expropriation of an artifact in one field to
solve problems in another field
 DSR projects in which the problem context is not well
understood but there exist mature artifacts in other fields that
can be exapted as effective solutions
 Exaptation DSR is judged by:
 Clearly grounding, representing, and communicating the exapted
artifact design
 Convincing evaluation providing evidence of how well the new
artifact solves the given problem
 All levels of artifact knowledge contribution can be made

101
Exaptation Exemplars
 Exaptation DSR is employed when new technologies provide
opportunities to solve new and/or different IS problems; for example:
 Codd’s exaptation of relational mathematics to the problem of database
systems design leading to relational database concepts, models, methods, and
instantiations (Codd, 1970; Codd, 1982)
 Berners-Lee original concept of the World Wide Web was one of simply
sharing research documents in a hypertext form among multiple computers.
In short time, however, many individuals saw the potential of this rapidly
expanding interconnection environment to exapt applications from old
platforms to the WWW platforms. These new Internet applications were very
different from previous versions adding many new artifacts to Λ knowledge
 Research by Berndt et al. (2003) on the CATCH data warehouse for health
care information. Well-known methods of data warehouse development (e.g.
Inmon, 1992) were exapted to new and interesting areas of health care
systems and decision-making applications

102
 103

Exaptation Exemplar – Adipat et al. 2011 MISQ

 Exapting effective web page presentation techniques to mobile devices
 Rigorous kernel theories from fit theory and information foraging theory
 Artifact – Presentation method
 Hybrid of tree-view, hierarchical text summarization, and colored keyword
highlighting
 Evaluation via prototype system
 Experimental design – Search tasks of varying complexity performed on five
variations of hybrid presentations
 60 university students
 Dependent variables
 Accuracy of search and time on task - measured
 Ease of use and usefulness – perception survey
 Research contributions
 Artifact improves effectiveness of web browsing on mobile devices
 Impact of task complexity on presentation exaptation
 Extends theories to mobile applications

103
Routine Design Quadrant
 Professional design or system building to be distinguished
from DSR
 However, evolving or best practices may be observed and
documented in “extractive case study” work (Van Aken)
 Study of best practices in routine design may lead to empirical
generalization
 Example – Davenport’s observation of BPR (Davenport & Short
SMR 1990)

104
 Papers mapped to Framework
Knowledge Article Knowledge Contribution Claims
Contribution
Improvement A Multilevel Model for Measuring Fit Between a Firm’s There is a need for a more fine-grained model for diagnosing the individual IS capabilities that
contribute to the overall fit or misfit between a firm’s competitive strategies and IS capabilities
Competitive Strategies and Information Systems Capabilities
(p.2) (See also Table 4).
(McLaren et al., 2011)
Improvement Guidelines for Designing Visual Ontologies to Support There could be several ways to address OWL’s inability to show state changes… We have taken a
different path, taking the view that we can keep the existing OWL syntax and improve the extent
Knowledge Identification (Bera et al., 2011)
to which it support s knowledge identification (pp. 885-886).
Exaptation Co-creation in Virtual Worlds: The Design of the User While Nambisan and his colleagues provide a useful framework for the online environment in
general, little is known about designing co-creation experiences in virtual worlds (p. 774).
Experience (Kohler et al., 2011)
 
Exaptation Design Principles for Virtual Worlds ABVWs comprise a new class of information systems… Thus, they require an extension of the
corresponding information system design principles (p. 675)
(Chaturvedi et al., 2011)
 
Improvement Correlated Failures, Diversification, and Information Security While our model to estimate security loss due to unavailable (i.e., system downtime) is based on
well-established queuing models, one innovation of our model is that the distribution from which
Risk Management (Chen et al., 2011)
the number of requests sent to the queue is drawn is endogeneous to system variables (p. 399).
 
Exaptation The Effects of Tree-View Based Presentation Adaptation on Presentation adaptation has been studied in the desktop environment and has been proven
beneficial … However, research on adaptation of Web content presentation for mobile handheld
Mobile Web Browsing. (Adipat et al., 2011)
devices is still rare (p. 100).
 
Improvement Improving Employees’ Compliance Through Information There is a need for IS security training approaches that are theory-based and empirically
evaluated. … (p. 757). To address this deficiency … this paper developed a theory-based training
Systems Security Training: An Action Research Study.
program … This paper then tested the practical workability through an action research
(Puhakainen and Sipponen 2010) intervention (p. 776).
 
Improvement Detecting Fake Websites: The Contribution of Statistical Systems grounded in SLT can more accurately detect various categories of fake web sites (p. 435).
Learning Theory. (Abbasi et al., 2010)
 
Improvement The Design Theory Nexus. ( Pries-Heje and Baskerville, The work suggests that the design theory nexus approach is more universal than previous
approaches to contingency theory, because it can operate in both symmetrical and asymmetrical
2008)
settings (p. 748).
Improvement Process Grammar as a Tool for Business Process Design. The method improves on existing approaches by offering the generative power of grammar-based
methods while addressing the principal challenge to using such approaches … (p. 757).
(Lee et al., 2008)
 
Improvement Making Sense of Technology Trends in the Information Our approach may complement existing technology forecasting methods … by providing
structured input and formal analysis of the past and current states of the IT landscape (p. 802).
Technology Landscape: A Design Science Approach.
(Adomavicius, et al., 2008)
Improvement CyberGate: A Design Framework and System for Text The results revealed that the CyberGate system and its underlying design framework can
dramatically improve CMC text analysis capabilities over those provided by existing systems (p.
Analysis of Computer-Mediated Communication. (Abbasi
811).
105 and Chen 2008)
Improvement Using Cognitive Principles to Guide Classification in Despite the importance of classification, no well-grounded methods exist .. (p. 840). We provide
empirical evidence…that the rules can guide the construction of semantically clearer and more
Thus
 In design science research
 Data - Using qualitative or quantitative data collection
methods
 Analysis
 Design and evaluate iteratively using descriptive
knowledge to create prescriptive knowledge
 Validity – Utility
To Summarize- Research Design
 It indicates, in addition to the process, possible output type (mainly
in DS)
 Starts from General approach to the details of every steps and
techniques
 Must answer the following fundamental questions with respect to
data and related issues– no reservation
 General approach
 Data needed: What are the data needed? –What are their nature?
Documentary? Statistical? Interview? Questionnaire?
 Data Source and sampling : Where are the data located?
 Data collection: How will the data be secured?
 Data analysis/experimentation/design and interpretation: How will
the data be analyzed and interpreted? How valid is the result from
the data?
One more thing
Conclusion

Good research results require a careful

design of the research methodology and
considerable evaluation efforts
Exercise
 Form a team (3-4)
 Select a potential research topic?
 Suggest possible research design/approach with justification
 Suggest specific research method/design
 Suggest data source, sample size and technique
 Identify type of data to be collected
 Identify method of data collection
 Identify method of data analysis
 Identify how to validate/evaluate your result

 Describe the procedure to conduct the research and represent it in a

form of flowchart