Developing a Questionnaire
Dr. Sadia Leghari
Using Questionnaires in Survey Research
Construction is key to valid and reliable research
Well written and manageable questions
– Manageable interviews
Consider why use survey methods?
Issues to consider
Getting clear response
Limiting response
Length
Intrusive
– personal questions and private thoughts
Procedure for Development
Specify what information will be sought
Determine the type of questionnaire and method of administration
Determine the content of individual questions
Determine the form of response to each question
Determine the wording of each question
How to Develop a Questionnaire
Guideline 1
Create a realistic list of all research questions and hypotheses
you wish to address.
List all the variables you need for the study.
This provides a checklist that can be used to make sure you
have items that address each research question or hypothesis.
Guideline 2 Include demographic, socioeconomic,
and geographic information.
You will need to control or adjust for variables such as
– race
– gender
– age
– income
– education
– marital status
– family size
– geographical information
List assumptions about participants
Language skills
What is their motivation to participate?
What are red flags for them?
Where might they misrepresent true beliefs?
What do they know?
What can’t they answer?
Questionnaires in other languages
Know your participants
Formal Spanish may not be ideal—mixed language
Translate to Spanish with appropriate adjustments.
Back translate to English using different translator.
Do practice interviews to find what is not clear—cognitive
mapping.
Cognitive Mapping
Ask person like those you will interview.
Person reads the question.
What is the question asking?
Person reads the response options and picks one.
Why did you pick that option?
Order counts
Order should be fixed so all participants answer questions in
same order.
Easy questions at the start.
Critical questions next in case somebody quits before the very
end of the survey.
Hard questions that are not absolutely critical at the end.
Sequencing can change answers.
Develop skip map
Computer assisted interviews where a person’s answer to one
item automatically routes them to the next appropriate item.
Written questionnaire limit skips.
Interviewer training is critical for skips.
Need to pretest with different types of people who will answer
different sets of questions.
Develop data entry plan
Computer assisted interviews automatic—produce dataset
ready to analyze.
Scanning in answers restricts format, but is accurate and quick.
Entering by hand is very time consuming and error prone.
– Step A—put numbers for each question in right margin.
– Step B—enter the data that is in the margins.
Forms
Self-administered surveys
Face-to-face interviews
Telephone surveys
Computer assisted and Web-based
E-mail
Self-Administered
Respondents complete on their own
Best designed for
– Measuring variables with numerous values or response categories
– Investigating attitudes and opinions not usually observable
– Describing characteristics of a large population
– Studying ‘private’ or ‘difficult’ behaviors
Self-Administered
Response rates tend to be lowest for mailed questionnaires
– As low as 20 – 30%
– Low response rate affects generalizability
More standardization of the questions
An increased reliability over other qualitative methods
– Response patterns can vary
Self-Administered
Closed-ended items limit the researcher
– Adjust for differences in respondents
– Clarify misunderstood items
– Explain ambiguity
Not suitable for all audiences
– Young children
– Visually impaired
– Learning/reading disabilities
Computer Assisted and Web-Based
Way to create and administer self-administered questionnaires
Marketing researchers find response rates increase
– This is not across the board
Questionnaires (especially short ones) can be sent via email
– Or provide internet link to site which hosts survey
Web-Based Surveys
Create own web page or hire a commercial company
– www.hostedsurvey.com
Allow for instant data coding
Need to be able to write code or use software
Maybe less time and costs
Access is a huge issue (affecting generalizability)
– Variation in computer ownership and usage
– Internet access
Interviews
Reading questionnaire items in a face-to-face or telephone
situation
Unstructured or in-depth
– Suited for exploratory research
– Either with one person or in focus groups
– Open-ended items
– Cannot standardize
– Good for complex situations
Interviews
Structured
– Consider role of interviewer
Style
Personal characteristics
– Influenced by the process
– Training is critical
– Follow wording
– Record responses
Example: General Social Survey (GSS)
Interviews
Response rates tend to be highest with face-to-face interviews
– Excluding ‘street corner’ technique
Issues
– More time
– Smaller samples
– Higher cost
Telephone
Most popular
– Less costly
– Less time
– Less subjective to interviewer
As compared to face-to-face
Often conducted with computers
Can probe for information/clarification
Threshold about 20 minutes
Constructions
Regardless of survey type, construct in a way
– Allows for candid answers
– Accurately
– Consistently
– Addresses goals, hypotheses, research questions
Conceptualizing the Task
List the set of research questions, hypotheses
– Consider how others have measured
– At least one questionnaire item for each variable
– Operationalize the concept
Outline what you want to cover
– Consider the following categories
Attitudes: feelings and opinions
Behaviors: what they actually do
Demographics: who they are
Measuring Behavior
Translate opinions into action
Useful in
– Making policy
– Improving working conditions
– Evaluation of programs
Measure of what they say they do
Instructions
Beginning of each section, should include clear
instructions
Disclose needed information for respondents to decide
whether to complete the survey
Consider phrasing – Don’t alienate
Tell whether
– Anonymous: no names or IDs
– Confidential: names or IDs, not attached responses
Instructions
Information about due date
– General time line is a good idea
Information on submitting
Be consistent with instructions and format
– A little variation might break up ‘routine’
Conceptualizing
Brainstorm
Reduce your list to items that are connected to your
‘research’ variables
Be creative and make an engaging survey
Length of questionnaire
– Time to construct
– Time for respondents to complete
– Cost of construction and dissemination
– # of variables/concepts
Font, spacing, and format
Question Writing
Target the vocabulary and grammar to the population be
surveyed.
– For studies within a specific organization, use the jargon used in that
organization.
– Be careful to avoid language that is familiar to you, but might not be to
your respondents. Avoid unnecessary abbreviations.
Types of Questions Illustrated
Precoded, single choice questions
4. What year of schoolare you in?
Freshman------------ -- 1
Sophomore---------- - - - 2
Junior-----------------3
Senior------------- - - -4
Single Choice Question
The population of the place I considered my
hometown when growing up was:
Rural area --------------------1
town under 5,000 --------------2
5,000 to 19,999 ---------------3
20,000 to 99,999 --------------4
100,000 to 999,999 ------------5
1,000,000 or over -------------6
Open-ended Questions
20. Approximately, what was your average in your final year of
high school?
21. What is (or was) your father's occupation? (e.g., foreman,
railway machine shop...supervises work of about 25 people.)
Job -------------------------------------------------------------------------
Open-ended Questions
In what year were you born?
Use open-ended when...
Too many response categories (yr. of birth)
you don’t wish to impose categories on respondent
“really” consulting respondent
qualitative--source of quotations
determining appropriate categorization
change-in-pace for respondent
Presence-absence questions
23. Have you ever had contact with handicapped in any of these
groups? (Circle to indicate "yes" or "no" for each group.)
Yes No
Community 1 0
Family 1 0
Relatives 1 0
Elementary school class 1 0
Junior high school class 1 0
Senior high school class 1 0
University class 1 0
As co-worker 1 0
Rank Ordering Questions
31. Rank order the three most important things you want in your job? (Place a 1 beside
the most important one; a 2 beside the next important one; and a 3 beside the next
most important one.)
High salary------------------- _____
Satisfaction------------------ ____
Continued interest----------- _____
Power------------------------- _____
Prestige---------------------- _____
Excitement-------------------- _____
Likert Style Questions
In the following items, circle a number to indicate the extent
to which you agree or disagree with each statement.
52. I would quit my present job if I won $1,000,000 through
a lottery.
Strongly Disagree 1 2 3 4 5 6 7 8 9 Strongly Agree
Tips for Likert Style Questions
“and” alert: avoid multidimentional
“strongly agree” always on right side
avoid double negatives--use direct negative statements
vary “strength of wording” to produce variation in response
make all respondents feel comfortable
Types of questions—closed ended
A. strongly agree, B. agree, C disagree, D strongly disagree.
Check income category. Check years of education. Check age
category.
Easy to enter data.
Easy to compare answers.
Types of questions—open ended
Participants own voice
Doesn’t impose researchers categorization
Confounded with language/verbal skills
Often skipped or trite answers—How is your marriage?
Answer—OK, I guess.
Structured questionnaires don’t have follow-ups to draw out
meaningful responses
Difficult to analyze
Vague/Double Meaning
Good questions involve one point or dimension.
Real bad: Has your spouse yelled at you or hit you in the last
month?
Problematic: How happy is your marriage?
– Happy regarding parenting
– Miserable regarding sexual relationship
– Great regarding equitable division of chores
– Terrible in terms of career goals
– Fantastic in terms of companionship
Be as specific and unidimensional as possible
Avoid value laden wording
Do you support the Tax Reform solution that will benefit so
many people including those on a fixed income, the elderly,
children, and the middle class?
Mitigate threatening questions
Have you smoked cigarettes in the last month?
Some people smoke. Did you happen to smoke in the last 12
months?
AVOID
Avoid ambiguity, confusion, and vagueness.
– Make sure it is absolutely clear what you are asking and how you want
it answered.
– Avoid indefinite words or response categories.
AVOID
Avoid emotional language, prestige bias and leading questions
– Watch out for loaded words that have a history of being attached to extreme situations.
– Watch for prestige markers that cue the respondent to give the "right" answer.
– Avoid leading questions.
– Avoid loading questions with extra adjectives and adverbs.
AVOID
Avoid double-barreled questions
– Make each question about one and only one topic.
Don't assume the respondent is an expert on
themselves (unless you have no choice)
AVOID
Avoid asking questions beyond a respondent's capabilities
– People have cognitive limitations, especially when it comes to memory
of past events.
– It is pointless to ask people about things that are not natural ways for
them to think.
AVOID
Avoid false premises
Avoid asking about future intentions (if you can)
Avoid negatives and especially double negatives
Formats: Grid
Response set—Reverse order--Format is efficient.
Item SA A D SD DK
1. I often feel depressed
2. I often feel I have no control
3. I control my own future
4. My boss appreciates the work I do
5. It is often difficult for me to sleep
Skip directions for self administered
Teenagers
1. Are you male or female?
Male (go to question 4)
Female (go to question 2)
2. Have you menstruated?
Yes (go to question 3)
No (go to question 4)
3. How much pain, if any, do you have when you menstruate the first time?
None
A little
Quite a bit
Extreme
4. How many days of school did you miss in the last month?
____ Enter the number of days
Pretest
Introduce--use blurb to introduce survey and to assure
confidentiality and anonymity
ease them into it
key and repeated questions at 1/3 point
group questions by type--don’t jerk around
anticipate computer data entry
