33 views

Uploaded by Mohamed Saad Bajjou

© All Rights Reserved

- 6 Fair empl.prac.cas. 1045, 6 Empl. Prac. Dec. P 8974 the Vulcan Society of the New York City Fire Department, Inc., Plaintiffs-Appellees-Appellants v. Civil Service Commission of the City of New York, Defendants-Appellants-Appellees, Nicholas M. Cianciotto, Intervenors-Defendants- Appellants-Appellees, 490 F.2d 387, 2d Cir. (1973)
- Guide to Six Sigma Statistics
- Research Methodlogy
- What is a Z Score and Hypothesis Testing
- ContentServer (1).pdf
- Diet Assessment Tool
- Size and the Structure of Authority in Canal Irrigation Systems.pdf
- situational judjement test - articol.pdf
- APAC2014.pdf
- Chapter 10
- Conway Lance 2010 _CMB
- GMA 2015 Meta-Analiza in Selectie Profesionala
- Importance of channels in Selling of Mutual Funds
- 2018 Muñoz & Cardona Translating in Fits and Starts Preprint
- IJMRA-RSS1676
- tests_of_significance_notes.pdf
- Session 5ee
- Serie3o
- Final report of Practical Training is the product of the concerted effort and help of a number people in make this report possible
- Week 5 - Chapter 1

You are on page 1of 1

Design research

measurement issues:

Operationalization issues The validity of the operationalization

Measurement issues Reliability, validity, sensitivity (see below)

Levels

(observed variables)

Independent

variables

Dependent

variables

(latent variables)

(latent variables)

Measures

Valid but

not reliable

Reliability threats

Valid and

reliable

Reliable but

not valid

Number of

developers

Stability reliability Does the measurement vary over time?

Representative reliability Does the measurement give the same answer when

applied to all groups?

Equivalence reliability When there are many measures of the same construct, do

they all give the same answer?

Development

team size

are not the focus of

research are irrelevant

variables.

Validity threats

Face validity Research community good feel.

Content validity Are all aspect of the conceptual variable included in the

measurement?

Criterion validity validity is measured against some other standard or measure for

the conceptual variable.

Predictive validity The measure is known to predict future behavior that is related

to the conceptual variable.

Construct validity A measure is found to give correct predictions in multiple

unrelated research processes. This confirm both the theories and the construct validity

of the measure.

Conclusion validity is concerned with the relationship between the treatment and

the outcome of the research 8choice of sample size, choice of statistical tests).

Experimental validity (see reliability)

H1

Developer

productivity

H2

Measurement relationship

associate latent variables

with their measures

Causal relationships (H1,H2) define

cause-effect relationship between

latent variables (theoretical

propositions). Can be tested only by

evaluating relationships between

observed variables (hypotheses)!

Represent

the effect

Represent

the cause

Measurement issues

Internal Is concerned with the validity within the given environment and the

reliability of results. It relates to validity of research process design, controls and

measures.

External Is the question of how general the findings are. Can you carry over the

research results into actual environment?

research instrument, for

example questionnaire.

Collect data

Describe abstract

theoretical concepts.

They cannot be

directly measured.

Developer

efficiency

Requirements

change

{OSSD, RUP,

XP}

Development

team size

Number of

developers

Latent

Observed

Software

reliability

LOC

Analyze data

Mean time

between failure

latent variables.

Each latent variable may

have multiple empirical

indicators.

How much does the measurement change with the change of the conceptual variable?

Qualitative

data

Use qualitative

data analysis

activity is to analyze

the collected data in

order to answer the

operationalized study

goal (research

question).

Quantitative

data

Chose data

analysis

Depends on data

and the goal of

the study.

Draw

conclusions

Consider reliability,

validity and sensitivity!

Consider

threats

Test

Real world

Simplify

observed?)

for the purpose of <Purpose> (what is the intention?)

with respect to their <Quality focus> (which effect is

studied?)

from the point of view of the <Perspective> (whose

view?)

in the context of <Context> (where is the study

conducted?).

Logic of sampling

to population

Sample

frame

Sampling

process

Sample

uses a random sampling process so that each

sampling element in the population will have an

equal probability of being selected.

A list of cases in a

population or the best

approximation of it.

Experiment design:

Define the set of tests

How many tests (to make effects visible)

Link the design to the hypothesis, measurement scales and statistics

Randomize, block(a construct that probably has an effect on response) and balance(equal number of subjects)

Random

assignment

Pretest

Posttest

Control

group

Classical

Yes

Yes

Yes

Yes

Yes

No

No

Yes

No

Yes

No

Yes

Yes

No

Yes

No

No

Yes

Yes

Yes

Yes

No

Yes

Yes

Yes

No

Yes

Yes

No

Yes

Experiment design

Experimental

group

Hypothesis

formulation:

Hypoth. statement

H0: positive

Ha: One or two tailed

Design

notation

o x o

R o

o

x o

o

x

x

x

o

o

o

o

o

x

ideas with others!

Construct validity

External validity

External validity

Experimental validity or

reliability

Experimental validity or

reliability

Construct validity

Experiment design

notation:

Investigate a phenomenon in depth, get close to the

phenomenon, provide a rich description and reveal

its deep structure.

enable the researcher to verify that findings are not

the result of idiosyncrasies of the research setting.

Cross case comparison allows the researcher to use

literal or theoretical replication.

and data collection may be undertaken

prior to definition of the research

questions and hypotheses.

Explanatory cases are suitable for doing

causal studies. In very complex and

multivariate cases, the analysis can make

use of pattern-matching techniques.

Descriptive cases require that the

investigator begin with a descriptive

theory, or face the possibility that

problems will occur during the project.

Discovery and induction:

Discovery is the description and conceptualization of the phenomena.

Conceptualization is achieved by generating hypotheses and developing

explanations for observed relationships

Statements about relationships provide the basis for the building of theory.

X = Treatment

O = Observation

R = Random assignment

o

Testing is concerned with validating or disconfirming existing theory.

Deduction is a means of testing theory according to the natural science

model

design components:

A study's question.

Its propositions, if any.

The unit of analysis.

The logic linking the

data to propositions.

The criteria for

interpreting the findings.

External validity

Repeated

ANOVA

ANOVA

Nom

3+

2

Int

We got an answer to

stated research question.

IV = Independent Variable

DV = Dependent Variable

3+

Linear

regression

The IV is the

variable that

defines

conditions

I nt +

I nt

Or

d

Or +O

d+ r d

In

t

S

The objective of this activity is to

report the study and its results so

that external parties are able to

understand the results in their

contexts as well as replicate the

study in a different context.

Case studies can be carried out by

taking a positivist or interpretivist

approach.

Can be deductive and inductive.

Can use qualitative or quantitative

methods.

Can investigate one or multiple cases.

Internal validity

Construct validity

A measure of central tendency is a single number that is used to represent the average score in

the distribution.

Mode the most common score in a frequency distribution

Median the middlemost score in a distribution

Mean the common average

Measures of variability

A single number which describes how much the data vary in the distribution.

Range The difference between the highest and lower score in a distribution.

Variance The average of the squared deviations from the mean.

Standard deviation the square root of the variance, a measure of variability in the

same units as the scores being described.

Correlation and regression

Determine associations between two variables.

Correlation The strength of the relationship between two variables.

Regression Predicting the value of one variable from another based on the

correlation.

Inferential statistics or statistical induction comprises the use of statistics to make inferences

concerning some unknown aspect of a population.

Sampling distribution has three important properties:

It has the same mean as the population distribution.

It has smaller standard deviation as the population distribution.

As the sample size becomes larger, the shape of the distribution approaches a normal

distribution, regardless of the shape of the population from which the samples are drawn.

Hypothesis testing - is the use of statistics to determine the probability that a given hypothesis

is true. The usual process of hypothesis testing consists of four steps.

Formulate the null hypothesis H0 (the hypothesis that is of no scientific interest) and the

alternative hypothesis Ha (statistical term for the research hypothesis).

Identify a test statistic that can be used to assess the truth of the null hypothesis.

Compute the P-value, which is the probability that a test statistic at least as significant as

the one observed would be obtained assuming that the null hypothesis were true. The

smaller the P-value, the stronger the evidence against the null hypothesis.

Compare the p-value to an acceptable significance value alpha (sometimes called an alpha

value). If p<=alpha, that the observed effect is statistically significant, the null hypothesis is

ruled out, and the alternative hypothesis is valid.

Statistical errors

The null

hypothesis

(H0)

H0 is TRUE

H0 is FALSE

Accept H0

Correct decision

Wrong decision

Type II error

Reject H0

Wrong decision

Type I error

Survey Process:

Study definition determining the goal of a survey.

Design operationalizing of the study goals into a set of questions (see theoretical model)

Execution the actual data collection and data processing.

Analysis interpretation of the data.

Packaging reporting about the survey results.

Ord

Here is the

distribution of

values of Z when

the hypothesis

tested is true.

(mean Z = 0)

Correct decision

of rejecting the

hypothesis tested when

that hypothesis is true.

Here we have

set alpha = 0.05

Friedman

Int

Paired

(related) ttest

Related

Ord

Int

Wilcoxon

Matched

Pairs

One way

ANOVA

Indep.

Kruskal Wallis

3+

Ord

Independent

t-test

Int

A survey is a study by asking (a group of) people from a population about their opinion on specific

issue with the intention to define relationships outcomes on this issue.

A smaller set of cases a

researcher selects from a

larger pool and generalizes

to the population

Population

Variables selection:

Independent and dependent variables

Observed variables

Measurement scales (nominal, ordinal, interval, ratio)

Selection of subjects:

Profile description

Quantity

Separation criteria

Does not explicitly control or manipulate variables.

Studies a phenomenon in its natural context.

Makes use of qualitative tools and techniques for data

collection and analysis.

Case study research can be used in a number of different

ways.

Can be used for description, discovery and theory testing.

Context selection:

Online vs. Offline

Student vs. Professional

Specific vs. general

Single

case

ify

Multiple

case

Reality

Abstract

/ Fa

ls

Internal validity

Internal validity

om

port

among respondents, association

and trend analysis, consistency of

scores.

Conclusion validity

S up

Literature survey.

to a representative

comparison baseline: sister

project, company baseline,

project subset with no

change.

Descriptive statistics are used to describe the basic features of the data in a study. They provide

simple summaries about the sample and the measures. Together with simple graphics analysis,

they form the basis of virtually every quantitative analysis of data.

Consider sources of

invalidity (internal,

external)

Disseminate

results

Questionnaires.

Interviews.

Project measurement.

Interviews

Parametric and

nonparametric

statistics, compare central

tendencies of treatments,

groups.

+

om

Mind

Operationalization

Easier to implement and collect data. Tick boxes.

Easier to make comparisons over time and between

places.

Can be a quick fix when organizations need

performance data to justify project investment.

Easier to process through a computer.

Easier for other stakeholders to examine and

comprehend.

Trends and patterns easier to identify.

Can distort the evaluation process as we measure

what is easy to measure.

Can lead to simplistic judgments and the wider

more complex picture is ignored.

Questionnaires.

Qualitative (Judgments)

Tends to be the poor relation.

Problems of opinion and perception when

making the judgment.

The data collected is more likely to create

differences of opinion over interpretation.

Not easily measurable.

As the benefits are longer term, they can be

outweighed by shorter term costs.

Can lead to inconsistent assessments of

performance between places over time and

between project elements.

Subjective opinions tend to be given less

status than quantitative ones.

World of propositions

measurement.

Questionnaires.

+

om er

N th

O

World of theory

measurement.

projects/organizations

keeping comparable data.

No control over variables

methods.

Can at most confirm

association but not causality.

Can be biased due to

differences between

respondents and

nonrespondents.

Questionnaire design may be

tricky (validity, reliability).

Use

quantitative

data analysis

statistically or analytically.

Positivism is a

philosophy that

states that the

only authentic

knowledge is

scientific

knowledge, and

that such

knowledge can

only come from

positive

affirmation of

theories through

strict scientific

method.

generalizes to many

projects/organizations.

Allow to use standard

statistical techniques.

Enable research in the

large.

Applicable to real world

projects in practice.

Generalization usually

easier.

Good for early exploratory

analysis.

With little or no

replication they may

give inaccurate results.

Difficult to interpret and

generalize (e.g., due to

confounding factors).

Statistical analysis

usually not possible.

Few agreed standards

on procedures for

undertaking case

studies.

is to run the study according

to the study plan.

Perform

research

population to which you want to

ultimately generalize results.

Application in

industrial context

requires

Compromises.

rval

Sensitivity

sample

I nt e

the same results under the same

conditions (consistency)?

Validity - does the measurement method

actually provide information about the

conceptual variable?

Sensitivity - how much does the

measurement change with the changes on

the conceptual variable?

Sources of invalidity

Design

survey

Design case

study

Can be incorporated in

normal development

activities.

Already scaled up to

life size if performed on

real projects.

Can determine whether

expected effects apply

in studied context.

Easy to plan.

Help answer why and

how questions.

Can provide qualitative

Insights.

Delphi method

Action research

Laboratory oriented methods

Mathematical modeling

Computer simulation

Laboratory experiment

Technology oriented methods

Proof of technical concept

Literature based methods

Literature review

Conceptual study

actually used in research.

Hypothesis testing

Hypotheses are tested by comparing

predictions with observed data

Observations that confirm a prediction do

not establish the truth of a hypothesis

Deductive testing of hypotheses look for

disconfirming evidence to falsify

hypotheses

Design

experiment

Other research

methods

Accessible population

population that you can actually

gain access.

Dependent

XP

per developer per day

Independent

RUP

...

(observed variables)

OSSD

Software

development

methodology

Select

research

method

relationships.

Can confirm theories.

Theoretical model is based on research question and represents set of concepts and

relationships between them!

Purpose

conceptualize the problem

stated in research question. It

is commonly represented with

causal model.

Pearson correlation

Linear regression

Create

theoretical

model

Spearman correlation

Linear regression

to demonstrate a familiarity

with a body of knowledge and

establish credibility. Additional,

to show the path to prior

research and how a current

research is related to it.

In

Does the proposed software improvement increases the efficiency of its users?

How does software development methodology and team size influence developers

productivity?

Review

Literature

Chi-squared

cross tabulation

Event

What?, Where?, Who?, When?,

How Much?, How many?, Why?,

How?

important part of any research.

Technology change is

implemented across a large

number of projects.

Description of results,

influence factors,

differences and

commonalities is needed.

One sample

t-test

Confidence in

theory is reduced

Theory is modified

Statistical methods can be used to summarize or describe a collection of data; this is called

descriptive statistics. In addition, patterns in the data may be modeled in a way that accounts for

randomness and uncertainty in the observations, and then used to draw inferences about the process

or population being studied; this is called inferential statistics. Both descriptive and inferential

statistics comprise applied statistics.

Change to be assessed

(e.g., new technology)

is wide-ranging

throughout the

development process.

Assessment in a typical

situation required.

va

l

Prediction is

NOT confirmed

which technology,

when, where, and

under which conditions

is possible.

To investigate self

standing tasks from

which results can be

obtained immediately.

t er

Laws are

formed

already known about a question

before trying to answer it.

Control

In

Relationships

between

constructs

are identified

theory development, research

design, data collection and data

analysis.

Major

threats

Level of access

Theory is

NOT modified

Define research

question

When

appropriate

Event

Research is

performed

you are trying to find out by

undertaking the research

process.

from a group of people, projects,

organizations or literature.

Investigate a typical

case in realistic

representative conditions.

Pro's

Theory is

formed that

explains laws

be defined here. Based on the

research question an empirical

strategy has to be chosen.

Establishes causal

relationships, confirm

theories.

Con's

Predictions

from theory

can be

drawn, which

form

hypotheses

Event

Theory is

rejected

Data

collection

Prediction is

confirmed

Analysis

types

Laws are

formed

explanation, and presentation of data.

Nominal

Event

Research is a systematic

process for answering

questions to solve

problems and create new

knowledge.

Nominal

getting at the correct

issues

But can you draw valid

conclusions?

Relationships

between

constructs

are identified

Start empirical

research

Empirical research is a

research approach in which

empirical observations (data)

are collected to answer

research question.

Discriminant of

Logistic Regression

Level of control

Explanatory

Why

research

How

How many

How much

Descriptive

research

When

Who

Where

Exploratory

research

What

Confidence

in theory is

increased

Chi-squared

Goodness of Fit

conclusions are valid ...

But are you getting at the

correct issues?

Data analysis

Coding scheme

(for open

question)

Data entry

Checking

Resolve

incomplete data

Statistical testing

of results

Ord

-3

-2

-1

Values of Zx

The Z score for an item, indicates how far and in what direction, that item deviates

from its distribution's mean, expressed in units of its distribution's standard

deviation.

Power is the probability

of rejecting the

hypothesis tested when

the alternative

hypothesis is true.

particular alternative hypothesis is true

(mean Z = 1)

Beta is the probability of accepting

the hypothesis tested when the

alternative hypothesis is true.

-3

-2

-1

Beta = 0.74

Power = 0.26

Values of Zx

Mann

Whitney U

of Z = 1.65

Bernd Freimut, Teade Punter, Stefan Biffl, & Marcus Ciolkowski 2002, State-of-the-Art in

Empirical Studies, Virtuelles Software Engineering Kompetenz-zentrum.

Johnston, R. & Shanks, G. Research Methods in Information Systems. 2003.

Neuman, W. L. 2005, Social research methods : qualitative and quantitative approaches,

5th ed. edn.

Winston Tellis 1997, "Introduction to Case Study", The Qualitative Report, vol. 3, no. 2.

www.wikipedia.org

Types of survey

Descriptive surveys are frequently conducted to enable descriptive assertions about

some population, i.e., discovering the distribution of certain features or attributes. The

concern is not about why the observed distribution exists, but instead what that

distribution is.

Explanatory surveys aim at making explanatory claims about the population. For

example, when studying how developers use a certain inspection technique, we might want

to explain why some developers prefer one technique while others prefer another. By

examining the relationships between different candidate techniques and several

explanatory variables, we may try to explain why developers choose one of the techniques.

Explorative surveys are used as a pre-study to a more thorough investigation to assure

that important issues are not foreseen. This could be done by creating a loosely structured

questionnaire and letting a sample from the population answer to it. The information is

gathered and analyzed, and the results are used to improve the full investigation. In other

words, the explorative survey does not answer the basic research question, but it may

provide new possibilities that could be analyzed and should therefore be followed up in the

more focused or thorough survey.

Question types

Open

Closed

Reporting response

rate

Total sample

selected

Number located

Number

contacted

Number returned

Number complete

Works 2.5 Slovenia License

Authors:

Gregor Polani

Email: info@itposter.net

University of Maribor

Faculty of Electrical Engineering and Computer Science

Institute of Informatics

Poster version: 0.6 (DRAFT)

http://researchmethods.itposter.net

- 6 Fair empl.prac.cas. 1045, 6 Empl. Prac. Dec. P 8974 the Vulcan Society of the New York City Fire Department, Inc., Plaintiffs-Appellees-Appellants v. Civil Service Commission of the City of New York, Defendants-Appellants-Appellees, Nicholas M. Cianciotto, Intervenors-Defendants- Appellants-Appellees, 490 F.2d 387, 2d Cir. (1973)Uploaded byScribd Government Docs
- Guide to Six Sigma StatisticsUploaded bymehdi810
- Research MethodlogyUploaded byKashif Niazi
- What is a Z Score and Hypothesis TestingUploaded byRohit Goyal
- ContentServer (1).pdfUploaded byKalpesh
- Diet Assessment ToolUploaded byme_sushmadahal
- Size and the Structure of Authority in Canal Irrigation Systems.pdfUploaded byEnrique Bellido
- situational judjement test - articol.pdfUploaded byAlina Ciabuca
- APAC2014.pdfUploaded byDe El Eurey Shine
- Chapter 10Uploaded byVenkata Kalyan
- Conway Lance 2010 _CMBUploaded byhafiz10041976
- GMA 2015 Meta-Analiza in Selectie ProfesionalaUploaded byCriss Hor
- Importance of channels in Selling of Mutual FundsUploaded byqwertyuiop_6421
- 2018 Muñoz & Cardona Translating in Fits and Starts PreprintUploaded byZogoibi
- IJMRA-RSS1676Uploaded byzaid chaudry1
- tests_of_significance_notes.pdfUploaded byManish K
- Session 5eeUploaded byasfadfafasasd
- Serie3oUploaded byFer Pereira
- Final report of Practical Training is the product of the concerted effort and help of a number people in make this report possibleUploaded byVingdeswaran Subramaniyan
- Week 5 - Chapter 1Uploaded byMaria Laarnie D. Mori
- Hypothesis and Research QuestionpppppUploaded byJoe Shewale
- Edu 618 Week 2 Journal.docxUploaded byKevin Smart
- POWER POINT PRESENTATION FOR RESEARCHUploaded byJohn Acosta Furuc
- Academic Word Sublist 4Uploaded byme_decent28457
- Nres2Uploaded byKimTot Octaviano
- Improving Maintenance Perception in Developed CountriesUploaded byChase Skilla
- Lilian - HypothesisUploaded byLilian Becas Salan
- Intro 2 3Uploaded byMario Bagesbes
- Chapter IIUploaded bynvze999
- BloomPolygon.pdfUploaded bySanjay Balwani

- 12 Week Lsat Study ScheduleUploaded byKen
- Chapter 1 Thinking Critically With Psychological Science, Myers 8e PsychologyUploaded bymrchubs
- CONSORT ChecklistUploaded byJoko Tri Wahyudi
- BRMUploaded byRashmi Ranjan Behera
- Water Filtration Lab RubricUploaded bycmillica1176
- smart plsUploaded byNoorashikin Edin
- KROES Design MethodologyUploaded bydariosandrone
- Mcq Testing of HypothesisUploaded byJosé Juan Góngora Cortés
- New Microsoft Word DocumentUploaded byItbar Khan
- Chapter 8 Homework Assignment Ver 1.0Uploaded bySashi Velnati
- Day 5-Guidelines in Preparing QuestionsUploaded byKimberly Ko
- cornell notes sci method studentUploaded byapi-261304144
- hypothesis testingUploaded byAlfie
- The statistical crisis in scienceUploaded bybiblos
- Proposed Action Research OutlineUploaded byObet Gonzales
- Individual Assignment Week 1 RES342Uploaded byMaria Esther Feliciano Delgado
- PSY315_Week1_Worksheet_1_1Uploaded byMatthewWeitz
- Training T TestUploaded byThothathiri Sadagopan
- ap statistics 2017-2018 term 3 assignment sohee hanUploaded byapi-369740044
- MA-Statistics-Activity-HYPOTHESIS-TESTING.docxUploaded byFritchie Sularte Paladen
- Ch.4 - Experimental RDUploaded byRahul Nair
- Assignment 2Uploaded byMohammad Ammad Ahmed Butt
- Oxford Psychology 3 and 4 2e Ch1 Research Methods in PsychologyUploaded byYucef Bahian-Abang
- Final ExamUploaded byCherry Ann Pascua Roxas
- Seminar 3.1 -Qualitative Intro & SkillsUploaded byVihang Naik
- AP Statistics Practice ExamUploaded byldlewis
- Syllabus in ResearchUploaded byNorberto Ebuen
- TOPIK_3Uploaded byNazlia Nadia
- example 6Uploaded byapi-341123529
- BRM 2 marksUploaded byShanmuga Priya