PART 6

DATA COLLECTION
1. WHAT IS DATA COLLECTION?
2. TRIANGULATION
3. CLASSIFICATION OF DATA SOURCES
4. CRITERIA FOR DATA ADMISSIBILITY
5. COMMON ERRORS IN DATA COLLECTION
6. INSTRUMENTATION
7. MEASUREMENT
8. SCALES
9. ERRORS
10. VALIDITY
11. RELIABILITY
12. PILOT TEST
 4
1 3
2 CLARIFICATION
OF CONCEPTS
REVIEW OF
IDENTIFICATION OF FOPRMULATION OF
RELATED
RESEARCH ISSUE/QUESTIONS RESEARCH PROBLEMS IDENTIFICATION
LITERATURE
OF VARIABLES

STATEMENT OF
HYPOTHESIS

EXPLORATORY
SELECTION OF SURVEYS
5 RESEARCH
DESIGN DESCRIPTIVE
CASE STUDIES

LABORATORY
CAUSAL EXPERIMENTS

FIELD
EXPERIMENTS

PARTICIPANT
OBSERVATION
NON-PARTICIPANT
SELECTION OF PRIMARY
6 DATA COLLECTION DATA
QUESTIONNAIRES
TECHNIQUES

INTERVIEWS
SECONDARY DIRECT
DATA COMMUNICATION
PROJECTIVE
 WHAT IS DATA COLLECTION?

- Decisions which the researcher makes regarding

the sources of information about the study.

- The researcher considers whether there are data

already available in one form or another or
whether he must gather new data.

2 TYPE OF DATA

1. PRIMARY

2. SECONDARY
 PRIMARY DATA
- New data can be gathered according to the
researcher’s own specifications in line with his
particular objectives.
- Generated by either questioning people thought
on the desired behaviour, or by observing
selected activities.
- Questioning through:
Mail Survey
Personal Interview
Focus Group
Delphi Technique
Projective Techniques
 SECONDARY DATA
Alternatively, there are many kinds of available
information that the researcher can used:
Public Records
Previous Studies
Company Documents/Annual Reports
Published Data
Government Documents
Trade Publications
Books

- Also may include the thoughts of non-observer.

 CRITERIA FOR DATA ADMISSIBILITY
- Be selective in choosing the data
- Screened secondary data for pertinent to
Research Question.
Screening process:
Data that are skewed should be noted when
presenting.
- Identify reasons for rejection or acceptance of
data
Reasons for Rejection:
- Relate to quality of data received and data
gathering process
- Data provided by an unqualified person.
 COMMON ERRORS IN GATHERING RESEARCH DATA

1. Pays insufficient attention to establishing and

maintaining rapport with the subjects. This often
leads to refusals to cooperate or to a negative
attitude that can reduce the validity of tests and
other measures.

2. Weakens the research design by making

changes for administrative convenience of the
population from which the samples are drawn.

3. Fails to explain the purposes of measures used

in the research to the subjects. If the subject
thinks a test or measure is silly or worthless, he
will not cooperate.
4. Fails to evaluate available measures
thoroughly before selecting those to be
used in the research. This often leads to
the use of invalid or inappropriate
measures.

5. Select measures of low reliability that

true differences are hidden by the errors
of the measurement.

6. Selects measurements that the

researcher is not qualified to use.
 TRIANGULATION
Defined: The combination of methodologies in the
study of the same phenomenon. May be based
upon:

- Collection of different kinds of data

- Multiple viewpoints allows for greater accuracy
- Validation process to ensure that the variance
reflected that of the trait rather than the method.

They might relate to the following:

Scale
Reliability
Convergent Validation
 …..Cont’d
Triangulation in Organizational
Research:
- Refers to the use of multiple methods
to examine the same dimension of a
research problem, eg.

The effectiveness of a manager may

be studied by interviewing, observing,
and by reviewing performance records.
TRIANGULATION… OTHER MEANING..

Within method: Uses multiple

techniques with a given method
to collect and interpret data, eg.

Use of multiple scales

Use of multiple group
comparison
INSTRUMENTATION
AND
MEASUREMENT
 INSTRUMENTATION AND MEASUREMENT

“If a variable exists, it exists in some amount. If it

exists in some amount, it can be measured” -
Thondike

Two principle issues confront all measuring

instrument:

Is it Reliable?
Is it Valid?
 INSTRUMENTATION
It is the process of selecting or developing
measuring devises and methods appropriate to a
given evaluation problem.

Specifically, instrumentation refers to:

- Questions to be asked in order to tap the

variables/concept
- To determine to what extent subjective
feelings, attitudes, or perceptions might exist in
different individuals.
- Developing or selecting instruments to
measure the responses.
 MEASUREMENT
It is the process of assigning numbers to
represent the properties of persons, objects,
events, or states.

These symbols are to have the same relevant

relationships to each other as do the things
represented.

The empirical definition if measurement does

not refer to the theoretical component of the
measurement process
Measurement considerations may help to
clarify the theoretical thinking and to suggest
new variables that are to be considered.

Careful attention to issues of measurement

may force a clarification of one’s basic
concepts and theories.

Measurement involves an explicit, or

organized plan for classifying (and
quantifying) the particular data at hand in
terms of the general concept in the
researcher’s mind.
 …Cont’d

This allows the concept to be used in the

analysis as a variable.

Systematic Analysis:
- Connection between system, Integration and
interaction.
- Researcher merely describe verbally a
succession of specific examples and indicate
the ways in which one property seems to affect
the other, or he may attempt to specify
systematically (using cross tabulation and
correlation) the degree to which one property
varies with the other
…..Cont’d

- In a complex system – which is made up of

parts.

- Researcher gathers data about the parts –

then add them up or piece them together in
some way.

- Research procedures that will show which

parts of the system contribute to the total
group pattern.
 CLASSES OF MEASURE
1.PHYSICAL TRACES
Very Indirect. Indicators of psychological and social
processes. Most prone to misinterpretation.

2. ARCHIVES
Documents and Records. Not to be taken at face
value.

3. SIMPLE OBSERVATION
Who spoke to whom. Position and spatial
arrangement. Facial and postural clues, Nonverbal cues.

4. HARDWARE
Photographs etc.
 COMMON ERRORS IN USE OF STANDARD
MEASURING INSTRUMENTS

1. Fails to check the content validity of measures in the

situation in which the research is to be carried out.

2. Fails to standardize or control data collection situation,

therefore introducing bias resulting from nonstandard
instruction, coaching of some of the respondent involved
in the study, and variations in degree of assistance given.

3. Checks overall validity and reliability of measures

selected but fails to check validity and reliability data on
subtest scores even though these scores are to be
employed in the research analysis.
4. Uses personality inventories and other self-reporting
devises in situations in which the subject might be expected
to fake replies In order to create a desired impression.

5. Assumes that standard measures what they claim to

measure without making a thorough evaluation of validity
data available.

6. Attempts to use measures that he/she is not sufficiently

trained to administer, analyze and interpret.

7. Fails to make optimum use of the testing time he has

available by administering long measurements when short
measurements are available that met the requirements of
the research project equally well.

8. Does not carry out a pretrial of his measuring instruments

and, as a result, make blunders in the administering
procedures during the collection of his first data, thus
introducing bias.
 SCALES
- A scale is an internally consistent plan for
developing a measure. The concern is to develop
a valid and reliable set of indicators of the
theoretical constructs.

- A tool of mechanism by which individuals are

distinguished based on the variables of interest.

- The type of scale achieved when we deputize the

numerical to serve as representatives for a state of
affairs in nature depends upon the character of the
basic empirical operations performed on nature.
 SCALES should composed of multiple items
rather than single item because:

- More likely to represent a complex theoretical

concept or attribute.
- Single-item measures lack precision because they
cannot discriminate among fine degrees of an
attribute.

- Single-item scales usually are less reliable than

multi-item scales
- Single-item scales provide only a single
measurement, preventing an assessment of their
measurement properties.
 FOUR BASIC TYPES OF SCALES
1. NOMINAL – Equality for placement in the classes.

2. ORDINAL – ‘greater than’ or ‘less than’ for objects.

3. INTERVAL – Equality of differences between

intervals.

4. RATIO –True zero point.

Each classes of scales:

Differentiate by the range of variance, and
Sets limit to the kind of statistical manipulation that
can legitimately be applied.
 NOMINAL/CATEGORICAL SCALE
Nominal Scale : y=f(x)
The objects are assigned to mutually exclusive labeled
categories, but there are no necessary relations among the
categories i.e. no ordering or spacing is implied if one entity is
assigned the same number as another, they are identical with
respect to a nominal variable. Otherwise, they are just different
e.g gender, geographic location, and marital status are
nominally scaled variables. The only arithmetic operation that
can be performed on such a scale is a count in each category.

PERMISSABLE STATISTICS INCLUDE;

Mode, contingency coefficient.

e.g. Book classification system

Phone numbers
 Cont’d… Nominal Scale
- Usually used to obtain personal data
- Qualitatively distinguish groups by
categorizing them
- Allows researcher to assign individuals into
certain groups (mutually exclusive/no
overlapping)
- Simple convenient category labels with no
intrinsic value
- Calculation of percentage/frequency
- Use mode as a measure of central tendency
- Any one-to-one correspondence commonly
called a Classification Scale.
 Cont’d….. Nominal Scale
Examples of Questions:

1. Gender: ___ Male ____ Female

2. Job Type
3. Educational Level:
4. Age Category:
 ORDINAL/RANK ORDER SCALE

Ordinal Scale: y=f(x)

This scale is obtained by ranking objects, or
arranging them in order with regard to some
common variable. The question is whether each
object has more or less of this variable than some
other object. However, no spacing is implied so
the permissible arithmetic operations are limited to
statistics such as the median or mode, but not the
mean. The finishing order in a horse race or class
standing illustrate this type of scale. Similarly,
brands of frozen vegetables can be ranked
according to that -- from highest to lowest.
 Cont’d…. Ordinal/Rank order…

- Rank order of the categories eg. according to

preferences
- Provide information at to how respondents distinguish
among the items on the list.
- Use median as a measure of central tendency
- Any strictly increasing function (to preserve order): A
ranking scale.

Permissible statistics include: Median, Percentiles,

Order correlation.
eg. Hardness, Pleasantness, better or less than.
 Cont’d…. Ordinal/Rank order…

Examples of Questions:
1. Rank the attributes below according to their importance in
influencing your choice of a restaurant: ( 1=Most Important; 5=
Least Important)
Attributes Rank
Price ____
Location ____
Pleasant Atmosphere ____
Service Quality ____
Food (tastes, Appearance) ____

2. Rank the cities below in order of suitability for the branch outlet:

Cities Rank
Ipoh ____
Johor Bahru ____
Penang ____
Kuantan ____
Seremban ____
 Cont’d… Ordinal/Rank Order

CHARACTERISTICS OF CATEGORY SCALE:

1. Limited resolution of categories:

- Loss of information
- Forced to make similarity-difference judgments
- May enhance measurement error.
2. Ordinal level of measurement.
3. Fixed number of categories affects response.
4. Responses are affected by:
- Order and spacing of stimuli
- Number of categories presented
- True range of stimuli
- Attempts to assign equally
- Number of stimuli presented for judgment
- Extreme category labels
- Frequency distribution of stimuli presented for judgment.
 INTERVAL SCALE
Interval Scale: y=a +bx, where b>0
Here, the numbers used to rank the objects also represent equal
increments of the attribute being measured. This means that
differences can be compared. The difference between “1” and
“2” is the same as “2” and “3”, but is only half the distance from
‘2’ and ‘4’. However, the location of the zero point is not fixed
since zero does not denote absence of the variable, rg.
Temperature. Farenheit and Celsius temperatures are
measured with different interval scales and have different zero
points. Interval scales have very desirable properties because
virtually the entire range of statistical operations can be
employed to analyze the resulting numbers, including addition
and subtraction. A recurring question of most attitude measures
is whether they are interval scales. Usually, it is doubtful that
the intervals between categories are exactly equal, but they may
not be so equal as to preclude treating it as an interval scale. A
good example is a ‘willingness to buy” scale with ten categories
labeled from 1 to 10. If this were an interval scale, we could say
that two people with scores of 2 and 4 respectively differed by
the same degree of ‘willingness’ as two other people with scores
of 8 and 10.
 Cont’d…. Interval Scale

- Indicate the magnitude of the differences in the

variables

- Allows the computation of the arithmetic mean

and use of measure of dispersion such as:
Standard Deviation
Variance
Coefficient of Variation
Product-Movement Correlation.

- This is a scale of equal differences, and has an

arbitrary zero point.
 Cont’d…. Interval Scale

EXAMPLE QUESTIONS
1. Using the scale below, please indicate your responses to each of the
item by circling the number that best describe your feeling:

STRONGLY DISAGREE NEUTRAL AGREE STRONGLY

DISAGREE AGREE

1. My job offers me a chance to 1 2 3 4 5

test my abilities
2. Mastering this job meant a lot 1 2 3 4 5
to me

All the time Sometimes Neutral Seldom Never

3. How frequently do you get 1 2 3 4 5

annoyed when you make
mistakes
4. How frequently do you get 1 2 3 4 5
annoyed when someone else
makes mistakes
 RATIO SCALES

Ratio Scale: y=cX, where c>0

This is a special kind of interval scale that has
a meaningful zero point. With such scales
eg. weight, market share, or dollars in saving
accounts; it is possible to say how many
times greater or smaller one object is than
another. A score of 4 is two times a score of
2. Obviously, attitude scales cannot achieve
this property.
 Cont’d… Ratio Scale…

- Measures the magnitude of the difference and the proportions in

the difference.
- Allows the computation of the arithmetic mean, and measure of
dispersion:
Standard deviation
Variance
Coefficient
- Increased fine tuning of the scales
- Use in organizational research when exact figures or objective
factors are rated.
- This is a scale of equal ratios and has an absolute zero.
- Permissible statistics include: geometric mean.
- Examples: Length, width, loudness scale, pitch scale.
Cont’d… Ratio Scale…

EXAMPLE QUESTIONS

1. How many other organizations have you worked

before joining this organization? _____

2. How many employees report to you? _____

3. How many years of experience do you have in the

current field? ____
 QUANTITATIVE JUDGMENT METHODS

- Assumes more than ordinal properties:

- Direct judgment: giving a numerical
rating or line estimates
- Limited response category rating scale
- Constant Sum: Distribute a given
number of points over alternatives
- Fractionation: provide a numerical
estimate of the ratio or stimuli
using specific standard.
CHARATERISTICS OF MAGNITUDE SCALES
1. Judgments are made relative to a reference.
2. Respondent is free to match the continuous
response measure to the perceived intensity of the
stimulus.
3. Respondent needs a training exercise.
4. Assumes a power function of a log function:
- Power Function: Subjects magnitude
estimates increases proportionately with
proportional increases in stimulus magnitude.
- Log Function: Geometric increases in
stimulus magnitudes are required to produce
arithmetic increases in subjective responses.
- Power Law: Equal stimulus ratios produce
equal subjective ratios.
 PROBLEMS OF SCALE FOR SOCIAL
SCIENCE/BEHAVIOURAL PHENOMENA

1. There is ambiguity of the stimuli and

the scalar dimensions.

2. There is a lack of known metric

properties of the stimuli. The true
value of the social or behavioural
stimuli is unknown.
 ATTITUDE SCALE
SCALE is essentially, a measuring devise
allowing the assignment of symbols or numbers
to individuals, or their behaviour.
By rule, such an assignment indicates the
individual’s possession of a corresponding
amount of whatever the scale is claimed to
measure.

- All methods of observation are inferential,

varying in the degree of objectivity they posses.
 TYPES OF ATTITUDE SCALES

1. Summated Rating Scales/Likert

Scales
2. Equal-Appearing intervals/Thurstone
Scales
3. Cumulative Scales/Quttman Scales
4. Semantic Differential Scales
 SUMMATED RATING/LIKERT SCALES

- Contain a set of items, all of which are considered

approximately equal in attitude or value loading.

- Individuals respond with varying degree of intensity

on a scale ranging between extremes such as:
Agree – Disagree; Like-Dislike; Accept-Reject

- The scores of the responses are summed and/or

averaged to yield an individual’s attitude score.
STEPS TO CONSTRUCT A LIKERT TYPE SCALE

1. Collect a large number of favourable and

unfavourable statements regarding the attitude
object.

2. Select from these approximately equal number of

favourable and unfavourable statements.

3. Administer these items to respondents.

4. Compute the score of each individual

5. Through item analysis, find the correlation between

the individual scores and the total scores of all
respondents
 LIKERT SCALE – AN EXAMPLE:

1. Capital punishment is a Strongly Agree Undecided Strongly Disagree

necessary penalty. Agree Disagree

2. Most students can Strongly Agree Undecided Strongly Disagree

handle an independent Agree Disagree
study program

3. School? Very Interesting Undecided Boring Very

Interesting Boring
 EQUAL-APPEARING INTERVAL/
THURSTONE SCALES

- A method for assigning specific scale values

to items representing different degrees of
favourable attitude.

- Each item is given a scale value - the

strength of the attitude.

- More difficult to construct than Likert-type

scale, yet yields similar results.
STEPS TO CONSTRUCT THURSTONE SCALES

1. Collect series of short, precise statements

reflecting attitudes toward a particular variable,
object or person.
2. An expert group sort the statements into series of
11 files according to their relative degree of
favourableness or unfavourableness.
3. For each item, plot the distribution of scaled values
(1 to 11) assigned by the various judges., locating
the median of this distribution.
4. Find the scale that should be assigned to each
statement value.
5. Eliminate items that have large discrepancies
among judges.
6. Select 20 to 30 statements that are spread out
evenly along the scale.
These statements become the attitude scale:

1. This school treats its students better than any other (10.2)***
schools

2. Doing it all over again, I’ll still teach at this school. (8.5)

3. The teachers and the students take advantage of each (5.0)

other.

4. If you do not have the influence around here, you are (2.3)
dead.

*** This is the scaled value for the item. The person responding to the
questions simply checks the items with which he agrees. The points
corresponding to those items are totaled and then divided by the number
of answered items to yield an average scaled value.
 CUMULATIVE SCALES/GUTTMAN
- Likert and Thurstone Attitude Scales contained heterogeneous
statements about various dimensions of an attitude.

- Guttman Scale consists of a relatively small set of homogeneous

items.

- Items can be arranged in order of difficulty, complexity (from low to

high) so that to answer correctly the last implies approved on all the
previous statements.

- When the scale is cumulative, we know the respondent’s total score.

Therefore, we can predict the answering pattern.

- Limitation – appropriate when a single clear-cut attribute is involved.

- However, a well-constructed scale may yield reliable measures of

psychological variables: tolerance, group conformity, etc.
 CUMULATIVE/GUTTMAN SCALE – AN EXAMPLE
I want to ask you some questions about a man known
to have been a convicted criminal and have served
time in jail.

1. Would you object to such a person living in your

community?
2. Would you object to him working where you are
employed?
3. Would you object to inviting him to your house?
4. Would you object to his marrying a member of your
family?
* Answering “Yes” to item 1 predicts “Yes” answers to
item 2,3 and 4. A “no” answer to item 4 predicts ‘No’
answers to items 3,2 and 1. A ‘No’ answer to item 1,
but a “Yes’ to item 2 predicts “Yes’ answers for items 3
and 4.
 SEMANTIC DIFFERENTIAL SCALE

- Based on the viewpoint that objects have two different

types of meaning for individuals:
i. Denotative - can be stated easily
ii. Connotative - can be measured indirectly.

- Meaning: Refers to the pattern of an individual’s ratings

of a certain object.

- Contain 3 elements:
i. Concept to be evaluated
ii. Polar adjective pair
iii. A series of undefined scale positions
(between 5 and 9)
 Cont’d…..SEMANTIC DIFFERENTIAL SCALE

- Osgood, Suci and Tannenbaum found 3

principal factors accounting for most of
the semantic loading:

i. Evaluative (eg. good – bad)

ii. Potency (eg. hard – soft)
iii. Activity (eg. Fast – slow)
 Cont’d…. SEMANTIC DIFFERENTIAL SCALE
POLAR ADJECTIVE PAIRS

Excerpts from Osgood’s Factor Analyzed List

FACTOR FACTOR FACTOR

LOADING LOADING LOADING
Evaluative Potency Activity

Evaluative: e:
Good Bad 1.00 .00 .00
Complete Incomplete .32 .05 .05
Sociable Unsociable .42 -.19 .18
Kind Cruel .52 -.28 .00
Clean Dirty .45 -.26 .02
Light Dark .38 -.30 .01
Graceful Awkward .38 -.23 .05
Pleasurable Painful .37 -.25 .07
Beautiful Ugly .52 -.29 -.02
Successful Unsuccessful .51 .08 .29
Important Unimportant .38 .04 .31
True False .50 -.03 .01
Positive Negative .48 .00 .07
Wise Foolish .57 .06 .11
interesting boring .40 -.09 .22
 Cont’d…. SEMANTIC DIFFERENTIAL SCALE

FACTOR FACTOR FACTOR

LOADING LOADING LOADING
Evaluative Potency Activity

Potency: .
Hard Soft -.24 .97 .00
Strong Weak .30 .40 .10
Severe Lenient -.25 .43 .04
Constrained Free -.16 .21 -.04
Constricted Spacious -.16 .26 .04
Heavy Light -.20 .48 -.02
Serious Humorous .01 .23 .09
Large Small .09 .21 -.05
Masculine Feminine -.14 .47 .03

Activity:
Passive .17 .12 .98
Active
Calm -.15 .03 .26
Excitable
Cold .12 .09 .26
Hot
Slow .01 .26 .35
Fast
simple .17 .05 .25
complex
Other Suggested Listings:
Understandability:
Predictable – unpredictable
Understandable – mysterious
Familiar – strange
Simple – complicated
Clear – confusing

Miscellaneous:
Traditional – progressive
Permissive – restrictive
Authoritarian – democratic
Structured – unstructured
Formal – informal
Flexible – rigid
Original – stereotyped
systematic- - unsystematic
Responsible – irresponsible
Easy – difficult
Open – closed
Relaxed – tensed
Tender – tough
Approach - avoid
EXAMPLE .. SEMANTIC DIFFERENTIAL SCALE:
Pairs of these polar adjectives are then selected according to the purposes of the
research and then arranged at opposite ends of a series of seven-point scale. The
concept to be rated is then presented at the top of the combined scale. For example:

Good _____ _____ _____ _____ _____ _____ _____ Bad

Slow _____ _____ _____ _____ _____ _____ _____ Fast
Large _____ _____ _____ _____ _____ _____ _____ Small
Ugly _____ _____ _____ _____ _____ _____ _____ Beautiful
Active _____ _____ _____ _____ _____ _____ _____ Passive
Light _____ _____ _____ _____ _____ _____ _____ Heavy
Clean _____ _____ _____ _____ _____ _____ _____ Dirty
Weak _____ _____ _____ _____ _____ _____ _____ Strong
Sharp _____ _____ _____ _____ _____ _____ _____ Dull
Delicat _____ _____ _____ _____ _____ _____ _____ Rugged
e
Dark _____ _____ _____ _____ _____ _____ _____ Bright
Round _____ _____ _____ _____ _____ _____ _____ Angular
ed
 TYPES OF ERRORS
HALO EFFECT: Tendency for an irrelevant feature to influence eg. first
impression

RATING ERROR: Over-rater, under-rater, central tendency.

HAWTHORNE EFFECT: Under study, novelty, modified environment,

knowledge of results.

SELF-FULFILLING PROPHESY: What the researcher ‘expects’.

‘JOHN HENRY’ EFFECT: Those in the control group are determined to

‘proof’ or to show the ‘experimental’ group.

PLACEBO EFFECT: Expectations and Suggestibility. Natural stimulus

given as it were the active treatment.

POST HOC ERROR: Assumption about cause-effect relationship.

LAW OF THE INSTRUMENT: Using a certain instrument as a means of

solving all’ problems.
 VALIDITY
- Is the instrument measuring what it claims to measure, is it relevant?

- Validity is defined as the extent to which any measuring instrument

measures what it is intended to measure – Concept of REALISM.

0 The Realism of a certain set of data consists of its

correspondence to some facts – its truth.
0 The Realism of a certain set of data consists of its connection
with some significant problem or with the purpose of the study -
its relevancy.

- “One validates not the instrument, but the interpretation of data

arising from the specified procedure”.

- The focus of validity should be to ensure that the measurement used

is related to an indication of the response expected, or the specific
criterion which it is supposed to predict.
 Cont’d… VALIDITY

Are we measuring the right concept?

Are we measuring the concept right?

These relate to the issue of

“AUTHENTICITY”

“ Can you realistically expect to answer the

Research Question as it is stated with the
Research Instrument being used?”
4. TYPES OF VALIDITY TO BE
CONCERNED WITH:

0 Statistical Conclusion Validity

0 Internal Validity
0 Construct Validity
0 External Validity
KEY QUESTIONS CORRESPONDING TO
VALIDITIES:

0 Is the relationship present?

0 If so, can the causal relationship be
attributed properly to the cited
independent variable and not any other?
0 What particular cause and effect
constructs are involved in the
relationship?
0 To what people, settings, and times can
the relationship be generalized?
STATISTICAL CONCLUSION VALIDITY

- Relates to statistical power and

effect size.

- Statistical power should be

estimated before data collection.
 TYPES OF VALIDITY
1. CONTENT VALIDITY: Does the content (items) adequately
i) Face Validity measure the concept? The more items,
the greater the validity.

2. CONSTRUCT VALIDITY: Does the instrument tap the concept as

expected
i) Convergent expected?
ii) Discriminant Some degree of agreed upon accuracy.

3, CRITERION-RELATED VALIDITY: Does the measure differentiate

in a manner that helps to
predict
i) Concurrent The scale discriminates the differences

4. EXTERNAL VALIDITY: The extent of generalization of the results

of a causal study to other field
setting.
 Cont’d… TYPE OF VALIDITY

I) FACE VALIDITY

- Used to indicate whether the instrument, in the face of

it, appears to measure what it claims to measure. Will
persons making use of the instrument accept it as valid
measure in the everyday sense of the word?

- It relates to the motivation of the responding individual.

- Reasons why he/she might provide inaccurate or

biased information:
0 Lack of motivation
0 Perceptual and cognitive limitations
0 Lack of crucial information
0 Inappropriate data elicitation methods.
 Cont’d ..TYPES OF VALIDITY
2. INTERNAL VALIDITY

- Addresses the question of whether the exploratory variables

themselves caused the observed results in the dependent variables
under study.

- Major threats to Internal Validity:

0 History – this refers to specific events that become
confounding variables during the total time period,
environmental change can occur over time.
0 Maturation - Occurs when the subjects change during the
study.
0 Testing Effects – First measurement influence the second
measurement. This may be due to “fatigue’
of the testing instrument.
0 Equivalency of Test Units - Test and control groups may not
receive equal influence of the effect of
explanatory variables upon the dependent
variables.
 Cont’d ..TYPES OF VALIDITY

3. EXTERNAL VALIDITY
- The extent of generalization of the results of a causal
study to other field setting.

- Three types:
0 Statistical Generalizability – this depends upon the
appropriate use of probability sampling procedures.
0 Conceptual Replicability or Robustness - Replicate
study should yield similar results.
0 Realism – whether the tasks, stimuli, and setting
employed were realistic. The more realistic, the more
generalizable the result.

GENERALIZABILITY
General relevance to a broad domain of tasks, setting, etc.
 Cont’d ..TYPES OF VALIDITY

EXTERNAL VALIDITY:
- Generalization can occur in 2 ways:
0 Sample to population, and
0 From the situation under study to other situations.

TYPES OF GENERALIZABILITY

1. Effect Application: Maps observed data directly into

events beyond the research setting
2. Theory Application: Effects observed in the research
are employed to assess the status of theory.
 RELIABILITY
- Is the measuring instrument accurate, consistent, and
stable?

- Reliability concerns with the extent to which any

measuring procedure yields the same results on
repeated trials (Time-Associated and Form-
Associated).

- Estimates of reliability are derived based on:

Average Correlation among Items (Internal
Consistency) in the measuring instrument.
 Cont’d…RELIABILITY
- It is identified with “truth”.
- Reliability is a necessary, but not a sufficient condition for
validity.
- Reliability of a measurement with respect to a given variable,
means the “constancy of its results” as that variable assumes
different values. The variables usually considered are:
0 The measuring event (eg. same person using the same ruler in
successive measurements of the same object).
0 The measuring instrument (rg. Different ‘forms’ of an intelligent test).
0 The person doing the measuring (eg. different eyewitnesses of the
same event).

- Repeated measures with the same instrument on a given

sample of data should yield similar results.
 3 KINDS OF RELIABILITY

1. STABILITY: Degree to which a process is invariant over

time, ie. Yields the same results at different points in
time.

2. REPRODUCIBILITY: Degree to which a process can be

recreated under varying circumstances, different
locations, involving different material forms, ie. Yields the
same results despite different implementations.

3. ACCURACY: Degree to which a process conforms in

effect to a known standard, ie. Yields the desired results
in a variety of circumstances.
 Cont’d…RELIABILITY
- Extent to which any measuring procedure yields
the same results on repeated trials.

STABILITY:
- The ability of a measure to maintain stability over
time despite uncontrollable conditions, and the
state of the respondents.

- Indicates the Stability and Consistency of the

measurement.

- Low vulnerability to changes in the situation. ***

Attest to the goodness of the measure.
 TWO TYPES OF STABILITY TESTS:

1. Test-Retest: Repeating the same test

on a second occasion.

2. Parallel-Form Reliability: Using two

measures for the same variable -
highly correlated.
 Cont’d…RELIABILITY

INTERNAL CONSISTENCY

- Indicative of homogeneity of the items in the

measure – “hang together as a set”.

TWO TEST OF CONSISTENCY

1. Inter-Item: Items are independent measure

of the same concept.
2. Split-Half; Divide the instrument into two
and compare the score.
 RELIABILITY:
POSSIBLE SOURCE OF VARIANCE IN A TEST SCORE

1. LASTING AND GENERAL CHARACTERISTICS

OF THE INDIVIDUAL.

1. General skills
2. General ability to comprehend instructions.
Testwiseness, techniques of taking tests
3. Ability to solve problems of the general type
presented.
4. Attitudes, emotional reactions or habits generally
operating in situations like a test situation
(eg. self-confidence)
II. LASTING AND SPECIFIC CHARACTERISTICS
OF THE INDIVIDUAL

1. Knowledge and skills required by

particular problems in the test.

2. Attitudes, emotional reactions, or

habits related to particular test stimuli
(eg. fear of high places brought to
mind by an inquiry such as fears on a
personality test)
III. TEMPORARY AND GENERAL CHARACTERISTICS OF THE
INDIVIDUAL
(systematically affecting performance on various tests at a
particular time).

1. Health, fatigue, and emotional strain.

2. Motivation, rapport with examiner.

3. Examples effects of light. heat and ventilation.

4. Level of practice on skills required by tests of this type.

5. Present attitudes, emotional reactions, or strength of

habits ( insofar as these are departures from the person’s
average or lasting characteristics eg. political attitudes during
an election campaign).
IV. TEMPORARY AND SPECIFIC CHARACTERISTICS OF
THE INDIVIDUAL

1. Changes in fatigue or motivation developed by this

particular test (eg. discouragement resulting from failure on
a particular item).

2. Fluctuations in attention, coordination, or standards of

judgment.

3. Fluctuations in memory for particular facts.

4. Level of practice on skills or knowledge required by this
particular test (eg. effect of special coaching).

5. Example: temporary emotional state and strength of habits,

related to a particular test stimuli (eg. question calls to mind
a recent bad dream).

6. Luck in the selection of answers by ‘guessing”

 PILOT STUDY
Study done in advance of the ‘big’ study to find out how the
respondents actually think or talk about the topic of interest.

Before a questionnaire is ready for use, it has to be pretested.

The purpose of the pretest is to check whether the ideas in
each question are clear to the respondent.

The pretest may show that some questions should be

reworded, or in the case of multiple-choice questions, different
or additional alternatives may have to be included.

The participants in the pilot should be similar to those who will

be included in the actual study.
If, after the pilot, a number of significant revisions are made, a
second pilot may be necessary.
 Cont’d.. PILOT STUDY

- Conducted on a very small sample.

- Testing to assure that the measuring

instrument has validity, and the questions are
understood and correctly understood.

- Check for syntax errors, general format

layout, and the comprehensiveness of the
instrument.
- Also, check for estimated time required for
responding.
 Cont’d.. PILOT STUDY

1. It permits a preliminary testing of the

hypotheses that leads to testing more
precise hypotheses in the main study. It
may lead to changing some hypotheses,
dropping some, and developing new
hypotheses when called for.

2. It often provides the researcher with ideas,

approaches, and clues not foreseen prior to
the pilot study. Such ideas and clues
greatly increase the chances of obtaining
clear-cut findings in the main study.
3. It permits a thorough check of the
planned statistical and analytical
procedures, thus allowing an appraisal
of their adequacy in treating the data.
Needed alterations also may be made
in the data-collecting methods, so that
data in the main study may be analyzed
more efficiently.
 Cont’d.. PILOT STUDY

4. It greatly reduces the number of treatment

errors, because unforeseen problems
revealed in the pilot study may be overcome
in redesigning the main study.

5. It may save the researcher a major

expenditure of time and money on a research
project that will yield nothing. Unfortunately,
many research ideas that seem to show great
promise are unproductive when carried out in
the field or laboratory. The pilot study almost
always provides enough data for the
researcher to make a sound decision on the
advisability of going ahead with the main
study.
6. In many pilot studies it is possible to get
feedback from research subjects and
other persons involved that leads to
important improvements in the main study
although the pilot study should follow the
main study procedure for the most part,
variations such as trying alternative
instruments and procedures and seeking
feedback of research are usually
desirable
 Cont’d.. PILOT STUDY

7. In a pilot study, the researcher may try out a number of

alternative measures, and then select those that
produce the best results for the main study without
some tentative evidence that they would be productive.

**If the student plans to continue beyond the Master’s

degree, the Master’s research may sometimes serve
as a pilot study for the later research to be carried out
as part of the doctoral program. The less research
experience the student has, the more he/she is likely to
profit from a pilot study. Because of this, the student
should attempt a pilot study whenever possible, even if
it must be limited to only a dozen or so cases.
 ASSIGNMENT 5

1. DETERMINE TYPES OF
DATA REQUIRED

2. SELECT DATA GATHERING

METHODS