Developing a Questionnaire

Chapter 4
Types of Questions
Open-ended
high validity, low manipulative quality

Closed-ended
low validity, high manipulative quality
Open-ended
An open-ended question is one in which
you do not provide any standard answers to
choose from.

1.How old are you? ______ years.
2.What do you like best about your job?
Closed-ended
A closed-ended question is one in which
you provide the response categories, and
the respondent just chooses one:

What do you like best about your job?
(a) The people
(b) The diversity of skills you need to do it
(c) The pay and/or benefits
(d) Other:
______________________________
Dichotomous Questions
Dichotomous Question: a question that has
two possible responses

Could be
Yes/No
True/False
Agree/Disagree
Questions based on Level of Measurement
Use a nominal question to measure a variable
Assign a number next to each response that
has no meaning; simply a placeholder.

Use an ordinal question to measure a variable
Rank order preferences
More than 5 10 items is difficult
Does not measure intensity
Interval Level
Attempt to measure on an interval level
Likert response scale: ask an opinion question
on a 1-to-5, 1-to-7, etc. bipolar scale
Bipolar: has a neutral point and scale ends are at
opposite positions of the opinion

Semantic differential: an object is assessed by
the respondent on a set of bipolar adjective pairs

Guttman scale: respondent checks each item
with which they agree; constructed as
cumulative, so if you agree to one, you probably
agree to all of the ones above it in the list

Filter/Contingency Questions
To determine if a respondent is qualified to
answer questions, might need a filter or
contingency question (also known as knowledge)

Limit # of jumps

If only two levels, use graphic to jump

If you can't fit the response to a filter on a
single page, it's probably best to be send them
to a page, rather than a question #
How many steps in the response scale?
Statistical reliability of the data increases
sharply with the number of scale steps up
to about 7 steps

After 7, it increases slowly, leveling off
around 11

After 20, it decreases sharply
Should there be a middle category?
Does it make sense to offer it?

Should not be used as the dont know or
no opinion option.

The middle option is usually placed between
the positive and negative responses.

Sometimes its last in an interview.
Direct Magnitude Scaling
Method of obtaining ratio-scaled data
Idea is to give respondents an anchor point,
and then ask them to answer questions relative
to that

Example:
Suppose you are interested in the severity of
crimes.
Begin by assigning a number to one crime
and then have respondents assign numbers
to the others based upon a ratio.
Filtering "Don't Know"
Standard format
No "don't know" option is presented to the respondent,
but is recorded if the respondent volunteers it.

Quasi filter
A "don't know" option is included among the possible
responses.

Full filter
First the respondent is asked if they have an opinion.
If yes, the question is asked.
Question Placement
It's a good idea to put difficult, embarrassing or
threatening questions towards the end
More likely to answer.

If they get mad and quit, at least you've gotten most of
your questions asked!

Put related questions together to avoid giving the
impression of lack of meticulousness

Watch out for questions that influence the answers
to other questions.
Wording of Questions
Direction of Statements
Response bias
Socially desirable

Always and never
Avoid this
Better to phrase as most, infrequently

Language
Reflect educational level and reading ability
Need for various languages

Frequency and Quantity
Consider both frequency and quantity

Consider number of times

Consider duration of times
Mutually Exclusive and Exhaustive
Mutually exclusive: not possible to select
more than one category/value

Exhaustive: providing all possible
categories/values
Forced Choice
Choose between 2 choices

Might not be relevant

Other choices exist (or at least possible)

Lesser of two evils
Recalling Behavior
Can be difficult to remember

Ask questions that can be answered

Choose time frames that are reasonable

Pilot test for time frame issues
Response Bias
Exaggerating the truth

Socially desirable answers

Consider using trap questions
Possibly fictional choice
Sensitive Items
More comfortable answering in categories

Minimize missing data

Might loose statistical power

Evaluating Questions
Pre-testing
Cognitive interviewing
Behavior coding
Peer review

Peer review has shown to be the best
method but its the least used.
Validity and Reliability Questions
Evaluative strategies:

Analysis of data to evaluate the strength of predictable
relationships among answers and with other
characteristics of respondents.

Comparisons of data from alternatively worded questions
asked of comparable samples.

Comparison of answers against records.

Measuring the consistency of answers of the same
respondents at two points in time.
Coding the Questionnaire
Create a codebook: reference guide for the
data set

Code: assigning a value to a response
category
Often numeric code
Pre-coding makes it easier
Content analysis on open-ended items
Yes/No often coded as present or not (0 or 1)
Missing Responses
Why blank?
Missed them
Refusal to answer
Didnt feel it applied
Didnt know the answer

To code or not
Analyze the difference
If know why, might consider
Piloting the Questionnaire
Test it on yourself
Possibly other experts

Test on people similar to sample
Dont reuse (some exceptions)

Discuss the survey with individuals
During completion or After
Finding Respondents
Best Methods of Selection

Even with a good survey, poorly chosen
sample leads to poor results