GVPT 479M Notes

Survey Methodology
- Study of the process of producing systematic surveys to help describe populations
accurately and test hypothesis
o Facts
o Opinions / Feelings & behaviors
- Interdisciplinary, though heavily influenced by psychology
- Considers and seeks to minimize numerous types of errors
o Think of error as any deviation from the truth

Basic Considerations
- Question design
o Question stem
o Answer choices
o Question order
- Data Collection
o Face to face
o Phone
 Live
 Interactive voice response (IVR)
o Self-Administered
 Mail
 Computer
o Combinations
- Sampling (How the sample is drawn)

Key Terms
- Population: group interested in studying
- Sampling Frame: list of elements used to draw sample, important for representativeness
- Sample: set of respondents selected to participate
- Probability sample: each person has a known probability of being selected into the
sample
- Sampling error: errors arising from leaving out some of the population (due to random
variation in the population)
- Cross-section: set of individuals interviewed once
- Repeated Cross-Section: draws new sample of individuals to be interviewed once with
same / similar questionnaire at different points of time
- Panel: set of individuals interviewed more than once
Error

Definitions not covered in detail:

- Trial: conceptual idea for repeated measurement if memory could be erased and survey
retaken multiple times
- Expected value: average value across all trials
- Nonrespondents: portions of sample who did not take the survey
- Unit/Item Nonresponse: missing answer for a given question by a respondent
- Imputation: statistical procedure for estimating values for missing data due to item
nonresponse
- Reliability: consistency of responses across repeated trials
- Response Variance: variability of responses across trials, leads to instability
- Undercoverage: parts of target population missing from sampling frame
- Ineligible units (over-coverage): elements of the frame that are not of the target
population

Keys to Inference from Surveys

- Answers accurately describe characteristics of respondents (i.e. attributes, behaviors)
- Sample is representative of the population of interest
- Deviations from either produces error

Types of Error
- Measurement Error (aka errors of observation)
o Differences between answers and concept being measure
o Leads to response bias: difference between expected value and true value
o Can be generated by poor question wording, misreporting (e.g. lie about
whether voted or not), memory failure, faulty records used to answer questions
- Errors of Nonobservation
o Differences between sample result (statistic) and true value in the population
 Random variation (sampling error)
 Don’t sample certain groups due to old/incomplete sampling frame
 College students living off campus during Census
 People without phones if survey done phone only
 Survey non-response (i.e. refusal to participate in survey or failed
contact)
Two Approaches
- Design Perspective
o Moves from abstract to concrete
- Quality Perspective
o Focus is on types of errors

Design Perspective
- Measurement Properties
o Construct: information sought
 o Measurement: how information (constructs) is obtained (the questions)
o Response: answers to the questions (can involve thought, memory, use of other
information, top of the head consideration)
- Representational Properties
o Target population: group of interest
o Frame of population: list of eligible portion of population that can be selected
into the sample
o Sample: portion of frame selected to take survey
o Respondents: those who agree to take survey
o Post-survey Adjustments: weighting data to help with representativeness

Quality Perspective (Total Survey Error Framework)

- Notation
o Subscript i is an index to track people, i=2, 3, 4,…N where N is the total size of
the population or sample
o u= value of a construct for person i in the population
o Yi= value of response to survey measurement for person i in the sample
o yi= value of response to survey measurement for person i in the sample after
editing and data processing
o ei= error for person i in the sample

Details for Quality Perspective

- Measurement Properties
o Construct Validity: how well our measure corresponds to the construct of
interest (Yi= ui + ei)
 Think about measuring age compared to measuring intelligence
 Think about measuring whether one voted compared to measuring
knowledge of politics
o Measurement Error: differences between answers and concept being measured
(yi-Yi)
o Processing Error: coding errors made by researches (yip-yi)
- Representational Properties
o Coverage Error: exclusion of portions of the target population from the frame
and possibly inclusion of people not in the target population (consider phone
survey)
o Sampling Error: errors due to not surveying the entire population
 Function of probability sampling stratification, clustering, sample size
o Nonresponse Error: errors due to some portion refusing to take the survey or not
being contacted
o Adjustment Error: errors due to weighting