The Questionnaire Development

Process
 Questionnaire
 Definition – A set of questions designed to generate
the information necessary to meet research objectives
 Characteristics
 Elicits information from respondents
 Results can be tabulated
 Standardized across respondents
 Understandable to respondents
 A good questionnaire must:
 provide the necessary information
 be considerate of respondents
 meet coding and data processing requirements
 The Questionnaire Design Process

Implementation
Implementation
Determine
Determine Data
Data
Collection
Collection Methods
Methods Determine
Determine Question
Question
Response
Response Format
Format

Prepare
Prepare Final
Final Copy
Copy
Determine
Determine Survey
Survey
Objectives
Objectives and
and
Constraints
Constraints Decide
Decide on
on Question
Question
Wording
Wording

Pretest
Pretest and
and Revise
Revise

Obtain
ObtainApproval
Approval Establish
Establish
from
from all
all Relevant
Relevant Evaluate
Evaluate the
the Questionnaire
Questionnaire Flow
Flow
Parties
Parties Questionnaire
Questionnaire and
and and
and Layout
Layout
Layout
Layout
 Step 1: Determine Survey Objectives

 Translates research objectives into information

requirements
 Survey objectives should be spelled out as
clearly and precisely as possible

 Rule 1 - Let the research objectives dictate

what questions to ask
 Rule 2 - Avoid unnecessary questions
 Step 2: Determine the Data Collection
Method
 We previously discussed the variety of ways that
survey data can be gathered
 Personal
 Telephone
 Electronic
 Mail
 Self administered
 Each method has a different impact on survey
design
 Step 3: Determine the Question
Response Format
Open-Ended Questions
 Open-ended questions are those in which the
respondent can reply in his/her own words
 Example of an open-ended question:

What are the advantages, if any, of ordering

from a mail order catalog compared with buying
at a nearby retail outlet?
 Open-Ended
Open-Ended Questions
Questions

Open-ended questions are useful:

 to interpret closed-ended responses
 when there are too many alternatives to list
 when alternatives vary between
respondents or are not clear
 when intensity of feeling is important
 when verbatim response is helpful
 for some sensitive issues (e.g. divorce)
 Open-Ended
Open-Ended Questions
Questions

Advantages
 More information

 Wide range of responses

 Maybe more accurate description

Limitations
 Respondent articulateness can lead to

error/variation
 Time consuming for respondent

 Coding and analysis is difficult

 Step 3: Determine the Question
Response Format
Closed-Ended Questions
 A closed-ended question is one that requires the
respondent to make a selection from a list of
options
 The primary advantage of closed-ended questions
is avoiding the problems of open-ended questions
(such as subjectivity, coding, etc.)
 May be dichotomous or multiple choice
 Closed-Ended Questions
 Types
 Dichotomous – Respondent chooses between

two alternatives
 Multiple-Choice – Respondent selects from

among several alternatives

 Example of a multiple-choice question:
Who do you think will win the World Series?
A. Tampa Bay Devil Rays
B. Boston Red Sox
C. Philadelphia Phillies
D. Los Angeles Dodgers
 Step 3: Determine the Question
Response Format
Scaled-Response Questions
 Question format that permits measurement of the
“intensity” of a respondent’s answers
 Example of a scaled-response question:
Now that you have tried the new sugar-free Slurpee,
would you say that you would… (Check one)
definitely buy it
probably buy it
might or might not buy it
probably not buy it
definitely not buy it
 Closed-Ended
Closed-Ended Questions
Questions
Advantages
 Easier to answer
 Reduce measurement error by overcoming
respondent articulateness and possible interviewer
bias
 Easy to code and analyze
 More likely to respond for some personal data
(e.g. income, age)
Disadvantages
 Information is lost
 Answers may not be precise
 Not able to develop alternatives
 Step
Step 4:
4: Question
Question Wording
Wording
 Rule 3: Use simple words
 e.g., not “marital status” but whether

“married,” “single” or “divorced”

 Rule 4: Use unambiguous words (words that
are easily and clearly understood)
 e.g. not “where do you usually buy school
supplies?”
but “from which retailer(s) did you buy school
supplies this year?”
 “where” and “usually” are often ambiguous
 Question
Question Wording
Wording
 Rule 5: Avoid “double-barreled” questions
(two questions in one)
 e.g., “What is your opinion about the price and
convenience of DART?”
 Rule 6: Avoid leading questions (questions
that imply an answer)
 e.g. “Do you agree that the government should
cut taxes so that we can lead better lives?”
 Question
Question Wording
Wording
 Rule 7: Avoid “loaded” questions (framing
questions so that they are leading or
emotionally charged)
 e.g., “Do you think chemical companies are doing
everything possible to control pollution?”
 Rule 8: Avoid implicit assumptions or
alternatives – State them clearly
 e.g., “Do you favor a law mandating methanol-free
gasoline” [if it means an increase in gas prices]
 Question
Question Wording
Wording
 Rule 9: Avoid burdensome questions (questions
that tax memory or skills)
 e.g., “How many different brands of breakfast
cereal have you bought in the last twelve months?”
 Rule 10: Clarify references (how should
respondent answer)
 e.g., “State your annual income” [personal income
or household income?]
 Question
Question Wording
Wording
 Rule 11: Alternatives should be mutually
exclusive, collectively exhaustive, and have
reasonable range
e.g., What is your annual household income?
__ $0 - $10,000
__ $10,000 - $15,000
__ $15,000 - $20,000
__ $40,000 - $60,000

What is wrong with this question and how

would you correct it?
Question
Question Wording
Wording –– Scaled
Scaled Responses
Responses
 Rule 12: Use appropriate phrases
e.g.,
B. Are you: (1) amazingly happy, (2) middling
happy or (3) kind of unhappy (check one)
A. Are you: (1) very happy, (2) happy or (3)
unhappy (check one)
 Rule 13: Avoid order bias
e.g., should you start with (1) as “very happy” or
(1) as “unhappy?”
 Question
Question Wording
Wording -- Sensitive
Sensitive //
Embarrassing
Embarrassing Questions
Questions
 Questions about borrowing money, personal
hygiene, sexual activity, criminal history, etc. must
be phrased carefully to minimize measurement error
 Suggestions/Techniques - Not Rules
 Ask non-threateningly and mix with other questions
e.g., How many times each day do you brush your teeth?
 Frame question to prompt more honest responses
e.g., Many people find it difficult to brush their teeth twice a
day. How often do you brush your teeth?
 Step 5: Questionnaire Flow and
Layout

After
After identifying
identifying
Use
Use screening
screening qualified
qualified respondent,
respondent,
questions
questions to
to Ask
Ask general
general
begin
begin with
with aa question
question
identify
identify qualified
qualified questions
questions first
first
that
that piques
piques the
the
respondents
respondents respondent’s
respondent’s interest
interest

Position
Position sensitive,
sensitive, Ask
Ask questions
questions that
that
threatening,
threatening, and
and Insert
Insert “prompters”
“prompters” require
require “work”
“work” in
in
demographic
demographic at
at strategic
strategic points
points the
the middle
middle of
of the
the
questions
questions as
as the
the questionnaire
questionnaire
end
end
 Step 5: Questionnaire Flow and Layout

 Rule 14: Opening questions should be simple

and interesting – begin with open-ended
question if warm-up is necessary
 Rule 15: Use “funnel approach” – ask most
general questions first, then increasingly
specific questions
 Rule 16: Questions should flow smoothly and
logically
Step 5: Questionnaire Flow and Layout
 Rule 17: Ask “screener” questions up front to
assess respondent qualifications
 Rule 18: Ask sensitive questions near the end
 Rule 19: Ask for demographic information at
the end
 Rule 20: The questionnaire’s appearance
should be attractive and professional
 Step 6: Evaluate the Questionnaire
 Is each question necessary?
 Is the questionnaire too long?
 Will the questions provide the desired information to
accomplish the research objectives?
 Is the layout attractive and professional?
 Step 8: Pretest and Revise

 Rule 21: Always pretest your questionnaire

Pretesting can be done by giving the questionnaire to
a few friends, “experts”, and potential respondents
(may be 5-10 persons total), asking them to fill out,
and obtaining their feedback
 Step
Step 8:
8: Pretest
Pretest and
and Revise
Revise

Pretest the questionnaire for:

 wording of individual questions

 flow of the questionnaire

 skip patterns

 length

 respondent interest and attention

 Step 9: Prepare Final Copy

 Even the final copy phase does not allow the

researcher to relax.
 Precise typing instructions, spacing, numbering,
and precoding must be set up, monitored, and
proofread,
before …

Step 10: Implement the Survey

 Other Considerations
 Questions should flow logically from one to the next
 General questions should be asked before more specific ones
 Earlier questions should not influence response to later ones
 Questions should flow from factual and behavioural questions to attitudinal and opinion
questions
 Questions should flow from the least sensitive to the most sensitive.
 Questions should flow from unaided (which brand do you prefer) to aided questions
(which brand do you prefer Tide, Cheer, Gain)

 Demographic questions should come at the end.

1. initial questions - screening and rapport questions

2. Middle - product specific questions
3. End demographic questions.
 Other Considerations
 Use multiple questions instead of one
 Similar questions together
 consistent mindset for respondents
 Develop a logical flow
 Use transitions between sections (E.g. In this section we ask questions about X)
 Distinguish between instructions, questions and responses
Filter and pivot questions should be used as necessary.

A FILTER question is one that screens out

respondents who are not qualified to answer a
second question.

A PIVOT question is a type of filter question that is

used to determine what version of a second
question to ask.
 QUESTIONNAIRE LAYOUT
The layout and physical attractiveness of a
questionnaire are important aspects
 Questionnaires should be designed to appear as
short as possible
 Questionnaires should not appear overcrowded

 Leave lots of space for open ended questions

 Questionnaires in booklet form are often

recommended
 OPENING
 Provide first name (at least)
 Provide name of company doing research
 Provide reason for survey and topic
 State that no selling will be involved and no personal data other than for statistical
purposes
 Tell respondent approximate time to complete
 Reinforce that respondent’s time is appreciated
 Invite to participate

AND CLOSING

 Thank for time

 Ask if they had a positive experience and remind them that their opinions count
 Pretesting and Correcting Problems

Purpose of pretest: To ensure that the questionnaire meets the expectations

in terms of the information that will be obtained
 Is question necessary – does it serve a purpose, will info be used
 Missing important variables – does it provide the info needed
 Match questions to objectives
 Pretest Specific Questions For
• Variation
• Meaning
• Task difficulty
• Respondent interest and attention
• Ambiguous, ill-defined, loaded, double-barreled questions
 Pretest the Questionnaire
• Flow of the questionnaire
• Skip patterns
• Length
 Respondent Interest and Attention
 Validity
 Validity refers to the question of whether our
measurements are actually hitting on the construct we
think they are
 While we can obtain specific statistics for reliability
(even different types), validity is more of a global
assessment based on the evidence available
 We can have reliable measurements that are invalid
 Classic example: The scale which is consistent and

able to distinguish from one person to the next

 Validity

Logical Statistical

Construct

Face Content Concurrent Predictive

Reliability Consistency Objectivity

 Types of Validity
 Criterion validity
 the degree to which the measure correlates with various outcomes

 Does some new personality measure correlate with the Big 5

 Concurrent Validity
 Criterion is in the present

 Measure of perception and current scholastic behavioral problems

 Predictive Validity
 Criterion in the future

 SAT and college gpa

 Construct-related validity
 How much is it an actual measure of the construct of interest

 Convergent Validity
 Correlates well with other measures of the construct

 Depression scale correlates well with other dep scales

 Discriminant Validity
 Is distinguished from related but distinct constructs
 Dep scale != Stress scale
 Reliability
 the consistency of your measurement
instrument
 the degree to which an instrument measures

the same way each time it is used under the

same condition with the same subjects
This clock reads 6:15
If nothing changes – if time
stands still, will the clock still
say the same thing?
YES! It’s very reliable. You
always know exactly what it
is going to say.
 Types of Reliability
 Relative Reliability
 Absolute reliability is the degree to which Relative reliability is the ... the differences
between the means of two sets
 Relative reliability is the degree to which individuals maintain their position in a sample over
repeated measurements.
 Absolute Reliability
 Absolute reliability is the degree to which Relative reliability is the degree to which repeated
measurements vary for individuals, individuals maintain their position in a ie the less they
vary, the higher the reliability in a sample over repeated measurements.
 Rater reliability (Objectivity)
 Intra-rater reliability
 In statistics, intra-rater reliability is the degree of agreement among repeated
administrations of a diagnostic test performed by a single rater.
 Interrater reliability.
 In statistics, intra-rater reliability is the degree of agreement among repeated
administrations of a diagnostic test performed by more than one rater.s