Field test options

instrument, instructions,
non-response/ refusals, and
Interviewer debriefing

Washington Group Regional Training Workshop

Rio de Janeiro
19 – 20 September 2005

Margie Schneider
HSRC, South Africa
 Recap
What we have covered so far:
– Purpose of the questions and where they came from
– Core set of questions and extended set and Q X Q
specifications
– Objectives of the test
– Translation protocol
What we still need to go through:
– How to design the test and related issues
– Enumerator training
– Plan for analysis and report writing
 Different components of the testing
process
• Pre-testing and expert review
• Testing internal validity on people with known
disability status (quota sampling and linked to
cognitive testing)
• Pilot testing – as for full Census or survey
• Full field test

Different sampling approaches will be discussed in

relation to different testing protocols
 Field Test options
• 2 questions to be answered:

1. What set of questions should be used in

field test?
2. What field test approach should be used?

Consider your own context, need, funding,

skills, etc. in deciding
 Question sets
1. Core set – 4 + 2 (6 domains of functioning):
Compulsory
2. Core + extended set on core domains : highly
recommended
3. Core + extended set + further additional
questions: nice to have for analysis
• 2 additional domains (learning and interpersonal
interactions)
• Psychological distress
4. Country specific question set : nice to have for
comparison with prior data
 What approach to testing?
• What factors to consider in deciding:
– Country needs
– Capacity to undertake test
– Funds available
– Time available
– Planned activities, e.g. other surveys
–?
 Census or national survey
• Use of question set in planned Census or
national survey
• Advantages:
– Large sample (5000 – 10 000 or whole pop)
– Minimal additional cost for disability questions
• Disadvantages:
– Not sufficient space for extended set
– Need to administer extended set and cognitive
test to sub-sample (costs of time and additional
training)
 Special study – small sample
• Select sample on basis of known ‘disability’
status
• Allows for construction of 2X2 table for
sensitivity and specificity calculations and
identifying true and false positives and
negatives – use of ‘gold standard’
• Suggest around 200 true positives and 200
true negatives
• Interviewer does not know status of
respondent (unless visible and obvious)
 Special study (contd.)
• Advantages:
– Larger set of questions
– Close observation of interview
– Cost is not exorbitant
• Disadvantages:
– Might not get full population representation
– Cost of establishing true status is high
– What counts as the true positive? (beyond the
easily observable)
– What is the ‘gold standard’?
 Special study – large sample
• Uses population based sample
• Sample size based on expected prevalence of
the different types of activity limitations
• Prevalence rates are usually low and so need
large sample to yield enough ‘disabled’
respondents
 Special study – large sample (contd)
• Advantages:
– Large data set
– Detailed set of questions - more than extended set
– Provides an indication of prevalence using core
and extended sets
• Disadvantages:
– Costly and time consuming
– Requires capacity to run and analyse
 Field test and cognitive test
• All respondents have core set
• One sub sample of respondents have
cognitive test (core + extended are
embedded)
• A second sub-sample of respondents do
– Core + extended
– Core + extended + further Qs
– Core + extended + country specific questions
• Some do all ?
 Non-responses and refusals
• How do you define these?
• How do you deal with these?

• Aim to:
– Reduce non-response and refusals
– Manage them when they do arise
 Reducing NR and refusals
• Use up to date sampling frame
• Clearly defined reasons for data collection
• Preparing the way – prior contact, letters, etc.
• Adequate interviewer training
• Allow budget for call-backs
• Plan for follow up of NR and refusals
• Separate refusals, part refusals, non-contacts
and sample loss (e.g. vacant dwellings)
 Effect of each on analysis
• Sample loss: does not create bias but reduces
sample size; need large enough initial sample
to take these into account
• Refusals and non-contact: bias where these
respondents might be different to those
respondents reached
• Need to understand reasons for refusals and
non-contacts – during pre-testing and pilot
stage
 Interviewer debriefing
• Interviewers are well placed to evaluate
process and content at an early stage
• Interviewers should note comments in
margins of questionnaires
• Different approaches:
– Group discussion (focus group technique)
– Interviewer rating forms
– Standardised interviewer questionnaires
– Combination of all three
 Enumerator training

Washington Group Regional Training Workshop

Rio de Janeiro
19 – 20 September 2005

Margie Schneider
HSRC, South Africa
 General points
• Select some disabled interviewers
• Importance of all interviewers having a good
understanding of what disability is and is not
and how it relates to the questions asked
• Avoid using term ‘disabled’ or ‘with
disabilities’
• All interviews are face to face
• Translation training
 General points (contd.)
• Confidentiality and understanding what this means
• Getting informed consent (ethics clearance?)
• Interviewers must be able to explain purpose of
survey
• Read questions in set order and with set wording
• Editing in field of completed questionnaires
• Submitting of completed interviews to head office
 Types of interviews
• Direct: respondent answers for him or
herself
• Interpreted: an interpreter ‘translates’ and
respondent answers directly
• Facilitated: a third party assists in explaining
(e.g. intellectually disabled person)
• Proxy: a person responds for another (e.g.
child)
 Interviewing disabled people
• Show respect and treat the person like
anyone else
• Don’t use first names unless permitted
• Address the person directly (not their
attendant)
• Ask how you can adapt your presentation to
make it easier (no need to ask what is wrong
with person)
 Hearing difficulties
• Lip reading
• Lighting
• Face the person
• Get person’s attention before speaking
• Reduce background noise
• Set the context – especially when changing
topics
• Use written communication (literate)
 Physical difficulties
• Accessibility of building where conducting
interviews
• Presence of attendant and confidentiality issues
• Get to same level (e.g. sitting for person using
wheelchair)
• Person to be seated comfortably
• Address person directly
• Pointing may be difficult
• Person may need breaks to move around
 Visual difficulties
• Large print and small print for cue cards
• Braille versions of cue cards
• Good contrast printing for pictures and print
(black on white or yellow)
• Identify yourself and others in the room
verbally
 Communication difficulties
• Clarify preferred mode of communication
• Repeat what you think was said to clarify
unclear speech
• Limit to yes / no questions
 Specific learning difficulties
• Manage problems in spatial orientation,
hand-eye coordination
• Limit auditory, visual and tactile distractions
• Avoid written text
• Explain carefully (if verbal language skills
are affected)
 Intellectual difficulties
• Be careful with informed consent
• Explain terms simply
• Listen carefully
• Have familiar person (friend or relative)
close by
• Use pictures or role play with little human or
animal figures
 Emotional or mental health
difficulties
• Side effects of medication
• Break up interview if too fatigued
• Give encouragement and support
• Manage expressions of frustration
• Manage stress
 Hidden difficulties
• Might not come forward with information
because of fear of stigma
• Effect of medication
• May need to break up interview