Session 20:

Data Collection in the Field

Ina F. Wallace
RTI International

IES/NCER Summer Research Training Institute, June 2007 1

 Data Collection in the Field
Topics Covered

• Methods of Data Collection

• Use of State-of-the-Art Data Collection
Procedures
• Ensuring Data Quality
• Conducting Small-Scale Validation Studies
• Ethical Issues in Data Collection

2
 Methods of Data Collection
• Interviewer Administered
– In Person
• Surveys
• Assessments
– Telephone
• Self-Administered
– Paper and Pencil
• Mail-in
• In-person
– Computer Assisted Self Interview
– Web procedures
3
 Data Collection Mode
Characteristics
High Data Collector Low Data Collector
Involvement Involvement
Paper Computer Paper Computer
Direct Face-to-Face Face-to-Face (Audio)
Contact with Paper and Computer Assisted Computer
Respondent Pencil Personal Interview/ Assisted Self-
Interviewing/ Assessment (CAPI) Interview
Assessment (CASI,
(PAPI) A-CASI)
Indirect Telephone Telephone Mail, Fax, E- Touch-tone
Contact with PAPI Computer Assisted mail Data Entry
Respondent Telephone Interview questionnare (TDE), Web,
(CATI) Electronic Mail
Survey (EMS)
No Contact Direct Computer Assisted Administrative Electronic Data
with Observation Data Entry (CADE) Records Interchange
Respondent (EDI)

4
Adapted From Biemer & Lyberg (2003). Introduction to Survey Quality
 Face-to-Face Interviewing
Advantages Disadvantages
• Maximum degree of •High cost due to travel
communication and •Can generate social
interaction desirability for sensitive
• Can utilize PAPI or CAI questions
• Allows for long, •Interviewers can affect
complex interviews respondents
• Variety of tactics can •Can generate
be used to gain interviewer variance
cooperation •Presence of others can
• Response rates affect responses
usually high •Falsification of
interview
5
 Telephone Interviewing
Advantages Disadvantages
• Cost lower than face to • Less flexible
face • No ability to use visual
• More easy to provide aids
training, supervision, • Must be shorter
and feedback • Questions must be less
• Social desirability bias complicated
less than with face to • Less capacity for
face motivating the respondent
• Interviewer variance • No coverage of non-
less telephone units
• Response rates lower
6
 Mail Surveys
Advantages Disadvantages
• Lowest cost • Little control over who
• Low social desirability completes the survey
• Can utilize visual aids • Item non-response high
• Allows respondent to • Response rates lower
take time and look up • Require a long field period
information (if to obtain an acceptable
necessary) response rate
• No interviewer variance • Respondents must have
good literacy skills
• Limit to closed-format
questions
7
 Computerized Procedures
• Computer Assisted Personal Interview
(CAPI)
• Computer Assisted Telephone Interview
(CATI)
• Computer Assisted Self Interview (CASI)
• Audio Assisted Self Interview (ACASI)
• Telephone Audio Assisted Self Interview
(T-ACASI)
• Computer Assisted Data Entry (CADE)
• Web Surveys
8
 Computer Assisted Procedures
Advantages Disadvantages
• Complex skip patterns and • Time consuming to develop
branching are easier to and program computer
follow • Must be adequately tested
• Previously acquired • Can’t change things easily
information can be once fielded
programmed • Risk of computer crashing
• Ranges, inconsistencies,
• Costly
and illegitimate values can
be checked
• Help screens can be
included
• Data are available for
analysis more quickly
9
 Direct Assessment
• Used to obtain psychological and
educational information about an individual
• Individually administered
• Research driven NOT for clinical or
diagnostic purposes
• Follows a standardized protocol

10
 Observational Techniques
• Involve a researcher making observations
within a natural setting
• Direct Observation - individuals know they
are being observed
– Continuous Monitoring recording as much
behavior as possible
– Time Allocation randomly visiting the sample
to observe at different times
• Unobtrusive Observation – individuals do
not know they are being observed
11
 Observational Techniques
Advantages Disadvantages
• No or limited • Variance associated with
participation needed observers is similar to
that found with
• May be good for interviewers
generating hypotheses • Can be very costly as
• Sensitive social issues they are often labor
can be obtained more intensive
accurately • Time consuming
• High in validity because • Reliability can be low
of the depth of • Generalizability can be
information collected low
12
 Summary:
Issues in Choosing a Strategy
• Type of Population
• Nature of Research Issue
• Question Form
• Question Content
• Response Rates
• Costs
• Length of Data Collection

13
 Methods of Administration
• Standardized Interviewing Approach
– A protocol in which interviewers interact with
the respondent in a manner which is
consistent across all interviewers
• Conversational Interviewing
– A protocol in which interviewers modify and
adapt questions to the respondent’s situation
• Hybrid Interviewing
– Combination of the two approaches

14
 Factors that Can Explain
Interviewer Variance
Interviewer Respondent
•Characteristics •Characteristics
•Appearance •Knowledge
•Motives •Interest/motivation
•Beliefs/attitudes •Confidence
•Perceptions •Strength of convictions
•Expectations •Expectations
•Skills/Knowledge

Questionnaire Survey Conditions and

•Definition clarity
•Terminology/jargon
•Question form
•Instructions
•Question wording
•Question topic
Interviewer Effects Setting
•Mode of interview
•Standardization
•Interviewer training
•Interviewer supervision
•Monitoring/observation

15
From Biemer & Lyberg (2003). Introduction to Survey Quality
 Factors that Can Explain
Interviewer Variance
Interviewer Respondent
•Characteristics •Characteristics
•Appearance •Knowledge
•Motives •Interest/motivation
•Beliefs/attitudes •Confidence
•Perceptions •Strength of convictions
•Expectations •Expectations
•Skills/Knowledge

Questionnaire Survey Conditions and

•Definition clarity Interviewer Effects Setting
•Terminology/jargon •Mode of interview
•Question form •Standardization
•Instructions •Interviewer training
•Question wording •Interviewer supervision
•Question topic •Monitoring/observation

From Biemer & Lyberg (2003). Introduction to Survey Quality 16

Role of the Interviewer/Assessor
• Gaining Cooperation
– Converting reluctant participants
– Recognizing and dealing with refusals
• Training and Motivating Respondents
– Interviewers implicitly and explicitly teach
respondents how to behave
• Obtaining Quality Data
• Ensuring Confidentiality

17
 Assessing Children
• Assessor Characteristics
– Flexibility
– Vigilance
– Self-awareness
– Eye contact
• Establishing Rapport
– Allaying apprehensions
– Helping children feel at ease
– Maintaining limits on permissible behavior
• Arranging the Physical Environment

18
 Ensuring Quality Data
• Recruiting interviewers/assessors
– Characteristics of all data collectors
– Characteristics of interviewers
– Characteristics of assessors
• Training on standardized procedures
• Certification as data collectors
• Monitoring performance

19
 Training
Content of Training
• Presenting the study
• Contacting participants
• Administering the
questionnaire/ assessment
• Probing
• Recording responses
• Confidentiality
• Interpersonal relations
• Administrative procedures
Procedures for Training
• Written materials
• Lecture and presentation
• Computer-based tutorials
• Planned exercises
• Role Playing
• Observation
• Certification
20
 Interviewer Training Agenda
• Introduction to the Study
• Role of the Interviewer
• Confidentiality Procedures
• Review of Questionnaire and Protocols
• Standardized Interviewing Techniques
• Round Robin Interview
• Gaining Cooperation
• Presentation of Scenarios
• Round Robin Demonstrations
• Paired Mocks
• Certification
21
 Assessor Training Agenda
Day 1: Day 2: Day 3:

Large group Small groups Small groups

• Introduction to Project • Individual practice with • Review of
• Role as an Assessor assessments assessments
• Best practices with • Paired practice with • Paired practice
children assessments • Post-training quiz

Small groups Large group Large group

• Pre-training quiz • Building rapport and • Concluding remarks
• Introduction to all gaining cooperation • Certification
materials • Administrative
• Learn each responsibilities
assessment • Data security
– Observation • Informed Consent
– Round Robins

22
 Monitoring Performance
What to Monitor How to Monitor
• Detection and prevention • Telephone verification
of falsified information with a sample of
• Compliance with rules respondents
and guidelines about • Direct observation – in-
administering the person or by telephone
questionnaire or • Audio-recording – CARI
assessment • Review of materials
• Performance on non- • Performance and
interview/assessment
production measures
tasks

23
 Human Subjects Issues
• Institutional Review Boards
• Informed Consent
– Who is carrying out research under whose sponsorship
– Description of purposes and procedures
– Duration of involvement
– Risks and Benefits
– Confidentiality
– Voluntary nature of research including stopping early
and skipping questions
– Compensation
– Contact information
• Confidentiality Issues
24
 Human Subjects Issues
• Institutional Review Boards
• Informed Consent
– Who is carrying out research under whose sponsorship
– Description of purposes and procedures
– Duration of involvement
– Risks and Benefits
– Confidentiality
– Voluntary nature of research including stopping early
and skipping questions
– Compensation
– Contact information
• Confidentiality Issues
25
 Data Collection Evaluation
Stage of Data Evaluation Method
Collection
Design Expert review of instrument
Pretest Behavior coding
Cognitive interviewing
Pretest/Post-Data Respondent focus groups
Collection Interviewer debriefings
Pretest/Data Supervisor observation
Collection Telephone monitoring
CARI
Post-Data Experimentation
Collection Internal Consistency
External validation
Re-interview/retest
Adapted From Biemer & Lyberg (2003). Introduction to Survey Quality
Purpose
Identify problems with
instrument
Evaluate one or more stages
of the response process
Evaluate instrument and data
collection procedures
Evaluate performance of data
collector
Compare alternative methods
of data collection
Evaluate components of MSE
Validate estimates
26
Post-Data Collection Evaluations
• Experiments
• Observational Studies
• Internal Consistency Studies
• External Validation Studies
• Administrative Record Check Studies
• Re-interview/Re-Test Studies

27
Post-Data Collection Evaluations
• Experiments
• Observational Studies
• Internal Consistency Studies
• External Validation Studies
• Administrative Record Check Studies
• Re-interview/Re-Test Studies

28