Survey Research

Week 5
10/7 & 10/9
Ch. 5
 Survey Research
• Used to assess people’s thoughts,
opinions, feelings
• Pragmatic
• Can have varying scope (broad/global
or narrow/specific)
• Correlational research
– Describe and predict
 Characteristics of Surveys
• Predetermined questions
– Oral, written, computer-entered responses
– Same questions given to all participants
• Representative Sampling
– Since same questions given, we can
describe general attitudes of the
sample/population
– Can compare/contrast attitudes of different
samples
Sampling in Survey Research
• Steps:
• Determine population of interest
• Decide who should respond
– Select representative sample
– Help us generalize findings
• We want to be able to describe/predict to greater
population, not just our respondents
– How representative our sample is depends on the
extent to which it exhibits the same distribution of
characteristics as the population
Sampling in Survey Research
continued...
• Population: all cases of interest (all DePaul
students)
• Sampling Frame: list of all members of
population of interest (registrar’s list)
• Sample: subset of population drawn from
sampling frame (50 students chosen for
study)
• Element: each member of a population (you)
 Sampling Bias
• Bias is a major threat to representativeness
• Selection Bias- when procedure used to
select the sample results in
over/underrepresentation of some segment of
the population
• Response Rate Bias- low/high response rate
among particular participants- may be biased
in some way (access, interest, educational
level, etc)
– Becomes almost like a convenience sample
 Sampling Approaches
• Nonprobability sampling- no guarantee
that each element has some chance of
being included in the sample
– Convenience Sampling- selecting
respondents based on availability-- Intro
Psych Courses, “person on the street,”
call-in surveys
 Sampling Approaches
continued...
• Probability sampling- each element has a
chance of being selected for the sample
– Superior to nonprobability sampling in ensuring a
representative sample
• Two types:
– Simple Random Sampling
• Every element has an equal chance of being included in
the sample
• Systematic sampling- choose every kth element
– Stratified Random Sampling
• Population divided into subpopulations (strata)- random
samples drawn from each strata
• Can draw equal random samples from strata or
 Survey Methods
• Mail Surveys
– Pros: self-administered, quick, eliminate observer
bias, anonymity of respondents
– Cons: no options for questions, low response rate
(response bias)
• Personal Interviews
– Pros: greater flexibility in asking questions, higher
response rate
– Cons: costly, interviewer bias (need to remain
neutral, do not write only selected portions of
response)
 Survey Methods continued...
• Telephone Interviews:
– Pros: 97% of households have phones
(representative), better access to dangerous
neighborhoods or people available only during
evening, can be completed quickly, better
supervision
– Cons: chance of selection bias and interviewer
bias, participants may respond differently to a
“faceless voice,” cell phones (“on the go”), higher
SES have multiple phone numbers, difficulty
remembering response options, telemarketers
 Survey Methods continued...
• Internet Surveys
– Pros: efficiency, cost, large/diverse samples,
cross-cultural, chat rooms (special interest or
support groups)
– Cons: potential for selection bias (internet access)
and response bias (lower response than
mail/phone), no real way to generate a random
sample of internet users, lack of control because
researcher is not present, ethical issues (informed
consent, protection from harm)
 Survey Research Designs
• Cross-Sectional Design
– One or more samples drawn from
population at one time
– Description- characteristics of a population,
differences between populations
– Prediction- predicting certain variables
using other variables (demographics)
 Survey Research Designs
continued...
• Successive Independent Samples
– Series of cross-sectional surveys conducted over
time (successively)
– Two key aspects:
• Same set of questions to all respondents
• Different samples drawn from the same population
– Describe changes in population over time
– Limitations
• Respondents do not necessarily “change their mind”-
different respondents
• Noncomparable successive samples- changes in
populations
 Survey Research Designs
continued...
• Longitudinal Design
– Same sample of respondents surveyed on numerous
occasions over time
– Can determine direction and strength of change over time
for the individual
– Possible to investigate reasons for attitude/behavior
changes
– Can assess effect of naturally occurring events (divorce,
maturation)
– Limitations
• Difficult to pinpoint exact reasons for change
• Difficult to find participants to survey over time-- attrition
– Threat to representativeness
• Possibility for reactivity, desire to remain consistent, reluctant to
report same problems/attitudes
 Questionnaires
• Most survey research uses questionnaires
• Measure various things:
• Demographic variables
– Accurate classification?
– Use deliberate and careful approaches
• Preferences and Attitudes
– Self-report scales
– Reliability and validity
 Reliability in Self-Report
Scales
• Reliability- consistency- yields consistent
results each time administered
– Scale must be reliable to make predictions
– Test-Retest reliability- administering same
questionnaire to large sample at 2 times
• Use correlations between participants scores (want r to
be at .80 or above)
– Reliability increases with larger number of
observations (in this case, questions/items)
– Test more reliable when administered to a diverse
sample-- more variability on factor being studied
– More reliable when testing situation free of
distractions, clear instructions are provided
 Validity in Self-Report Scales
• Validity- truthfulness- item measures what it
intends to measure
• Construct Validity- measures theoretical
construct it was intended to measure
– Convergent validity- two measures are correlated
(converge) when measuring a particular construct
– Discriminant validity- measure discriminates
between two different theoretical constructs
constructs (intelligence and knowledge of a certain
subject)
 Constructing a Questionnaire
• Steps for Constructing a Questionnaire
– What information should be sought?
– Predict likely results- will it answer your question?
– What type of questionnaire should you use?
– Survey method? Develop own or use an established
scale?
– If using established, can skip later steps
– Write first draft
– Reexamine and revise
– Have friends and experts revise
– Pretest questionnaire
– Under similar conditions, ask for feedback
– Edit questionnaire and specify
procedures/instructions for its use
 Guidelines for Effective
Wording of Questions
• Respondents believe wording of a question is obvious
– Interpret ambiguous/vague words based on presumptions
• Respondent assumes researcher will ask them
questions they can answer questions (even if there is
no valid answer)
• Vocabulary should be simple, direct, and familiar to ALL
participants
• Use short questions (<20 words)- all conditional info
precedes key idea
• No “double-barreled” questions (write separate
questions)
• No leading questions (leads respondent; mention all
possible perspectives or none) or loaded questions
(contain emotion-laden words)
 Guidelines for Effective
Wording of Questions
continued...
• Two general types of questions
– Free-response (open-ended)
– Respondents give own answers
– Allows for greater flexibility
– Needs extensive coding/responding
– Closed (multiple choice)
– Answer choices provided by researcher
– Can be answered quickly/easily
– Easier to summarize/compare responses
– Reduce spontaneity and “respondent’s voice”
– May choose less-than-preferred response
 Ordering of Questions
• Self-administered
– Start with interesting questions, end with
demographics
• Personal/Phone Interview
– Start with demographics, move into harder
questions
• Funnel questions- start with general, move to
specifics
• Filter questions- general questions that can
allow respondents to skip unnecessary
questions (mainly to be used for objective
information-- “Are you a DePaul student?”)
 Things to Keep in Mind about
Survey Research
• Truthfulness of participants’ responses
– Reactivity, social-desirability
• Be aware of their existence!
• When two variables items are correlated we can
make predictions for the variables/items
– “Correlation does not imply causation”
– Causation could go either way
– Spurious variables- third variable that accounts for
relationship (may or may not be measured)
– Mediating Variables (explains correlation between two
variables) and Moderating Variables (affect
direction/strength of correlation)-- found through Path
Analysis
Example of Mediator Variable

Stress

Poverty Psychopathology
 Example of Moderator
Variable
Stress Level

Time Studying Performance on Midterm

100
90
80
70
60
High Stress
50
Low Stress
40
30
20
10
0
Low High
Study Study
 Midterm Exam Tuesday
(10/14)
• Worth 50 points
• 30 multiple choice/true-false/matching
questions (1 point each)
• 5 short answer (2 points each)
• 2 long (paragraph) answer (5 points
each)
• Anything from notes and readings is fair
game
 Group Exercise (back-up):
Get into Groups of 4
• Construct a questionnaire-- stress during
exams/studying habits
• Steps of constructing a questionnaire (information
sought, type of questionnaire, draft, revise, pretest,
edit and specify procedures)
• Clearly worded, short questions- no double-barreled,
leading, or loaded questions
• Ordering of questions? Self-administered versus
interview?
• Closed/open-ended questions?
• What type of sampling? Nonprobability (convenience)
versus probability (simple random, stratified random)
 Class Exercise for Thursday
• Fill out this form
• Be sure to print out the “Thank You”
page
– Worth 5 extra credit points, but MUST bring
it to class on Thursday for credit
• We will go through ways of how to
analyze data using survey forms
• Midterm Evaluation for ME
• Please fill out this form, and be honest (I will
not be able to see your responses)
• Be sure to PRINT the “Thank You” page at
the end for credit (this assignment will be
worth 5 extra credit points- MUST be turned
in during class time either Thursday or next
Tuesday)