Developing a Questionnaire

Dr. Sadia Leghari

Using Questionnaires in Survey Research

 Construction is key to valid and reliable research

 Well written and manageable questions

– Manageable interviews

 Consider why use survey methods?

Issues to consider

 Getting clear response

 Limiting response

 Length

 Intrusive
– personal questions and private thoughts
Procedure for Development

 Specify what information will be sought

 Determine the type of questionnaire and method of administration

 Determine the content of individual questions

 Determine the form of response to each question

 Determine the wording of each question

How to Develop a Questionnaire

 Guideline 1
 Create a realistic list of all research questions and hypotheses
you wish to address.
 List all the variables you need for the study.
 This provides a checklist that can be used to make sure you
have items that address each research question or hypothesis.
Guideline 2 Include demographic, socioeconomic,
and geographic information.

 You will need to control or adjust for variables such as

– race
– gender
– age
– income
– education
– marital status
– family size
– geographical information
List assumptions about participants
 Language skills
 What is their motivation to participate?
 What are red flags for them?
 Where might they misrepresent true beliefs?
 What do they know?
 What can’t they answer?
Questionnaires in other languages

 Know your participants

 Formal Spanish may not be ideal—mixed language
 Translate to Spanish with appropriate adjustments.
 Back translate to English using different translator.
 Do practice interviews to find what is not clear—cognitive
mapping.
Cognitive Mapping

 Ask person like those you will interview.

 Person reads the question.
 What is the question asking?
 Person reads the response options and picks one.
 Why did you pick that option?
Order counts

 Order should be fixed so all participants answer questions in

same order.
 Easy questions at the start.
 Critical questions next in case somebody quits before the very
end of the survey.
 Hard questions that are not absolutely critical at the end.
 Sequencing can change answers.
Develop skip map

 Computer assisted interviews where a person’s answer to one

item automatically routes them to the next appropriate item.
 Written questionnaire limit skips.
 Interviewer training is critical for skips.
 Need to pretest with different types of people who will answer
different sets of questions.
Develop data entry plan

 Computer assisted interviews automatic—produce dataset

ready to analyze.
 Scanning in answers restricts format, but is accurate and quick.
 Entering by hand is very time consuming and error prone.
– Step A—put numbers for each question in right margin.
– Step B—enter the data that is in the margins.
Forms

 Self-administered surveys

 Face-to-face interviews

 Telephone surveys

 Computer assisted and Web-based

 E-mail
Self-Administered

 Respondents complete on their own

 Best designed for

– Measuring variables with numerous values or response categories
– Investigating attitudes and opinions not usually observable
– Describing characteristics of a large population
– Studying ‘private’ or ‘difficult’ behaviors
Self-Administered

 Response rates tend to be lowest for mailed questionnaires

– As low as 20 – 30%
– Low response rate affects generalizability

 More standardization of the questions

 An increased reliability over other qualitative methods

– Response patterns can vary
Self-Administered

 Closed-ended items limit the researcher

– Adjust for differences in respondents
– Clarify misunderstood items
– Explain ambiguity

 Not suitable for all audiences

– Young children
– Visually impaired
– Learning/reading disabilities
Computer Assisted and Web-Based

 Way to create and administer self-administered questionnaires

 Marketing researchers find response rates increase

– This is not across the board

 Questionnaires (especially short ones) can be sent via email

– Or provide internet link to site which hosts survey
Web-Based Surveys
 Create own web page or hire a commercial company
– www.hostedsurvey.com

 Allow for instant data coding

 Need to be able to write code or use software
 Maybe less time and costs
 Access is a huge issue (affecting generalizability)
– Variation in computer ownership and usage
– Internet access
Interviews

 Reading questionnaire items in a face-to-face or telephone

situation
 Unstructured or in-depth
– Suited for exploratory research
– Either with one person or in focus groups
– Open-ended items
– Cannot standardize
– Good for complex situations
Interviews

 Structured
– Consider role of interviewer
 Style
 Personal characteristics
– Influenced by the process
– Training is critical
– Follow wording
– Record responses
 Example: General Social Survey (GSS)
Interviews

 Response rates tend to be highest with face-to-face interviews

– Excluding ‘street corner’ technique

 Issues
– More time
– Smaller samples
– Higher cost
Telephone
 Most popular
– Less costly
– Less time
– Less subjective to interviewer
 As compared to face-to-face

 Often conducted with computers

 Can probe for information/clarification

 Threshold about 20 minutes

Constructions

 Regardless of survey type, construct in a way

– Allows for candid answers
– Accurately
– Consistently
– Addresses goals, hypotheses, research questions
Conceptualizing the Task

 List the set of research questions, hypotheses

– Consider how others have measured
– At least one questionnaire item for each variable
– Operationalize the concept

 Outline what you want to cover

– Consider the following categories
 Attitudes: feelings and opinions
 Behaviors: what they actually do
 Demographics: who they are
Measuring Behavior

 Translate opinions into action

 Useful in
– Making policy
– Improving working conditions
– Evaluation of programs

 Measure of what they say they do

Instructions

 Beginning of each section, should include clear

instructions

 Disclose needed information for respondents to decide

whether to complete the survey

 Consider phrasing – Don’t alienate

 Tell whether
– Anonymous: no names or IDs
– Confidential: names or IDs, not attached responses
Instructions

 Information about due date

– General time line is a good idea

 Information on submitting

 Be consistent with instructions and format

– A little variation might break up ‘routine’
Conceptualizing
 Brainstorm
 Reduce your list to items that are connected to your
‘research’ variables
 Be creative and make an engaging survey
 Length of questionnaire
– Time to construct
– Time for respondents to complete
– Cost of construction and dissemination
– # of variables/concepts

 Font, spacing, and format

Question Writing

 Target the vocabulary and grammar to the population be

surveyed.

– For studies within a specific organization, use the jargon used in that
organization.

– Be careful to avoid language that is familiar to you, but might not be to

your respondents. Avoid unnecessary abbreviations.
Types of Questions Illustrated

 Precoded, single choice questions

 4. What year of schoolare you in?

Freshman------------ -- 1
Sophomore---------- - - - 2
Junior-----------------3
Senior------------- - - -4
Single Choice Question

The population of the place I considered my

hometown when growing up was:

Rural area --------------------1

town under 5,000 --------------2
5,000 to 19,999 ---------------3
20,000 to 99,999 --------------4
100,000 to 999,999 ------------5
1,000,000 or over -------------6
Open-ended Questions

20. Approximately, what was your average in your final year of

high school?

21. What is (or was) your father's occupation? (e.g., foreman,

railway machine shop...supervises work of about 25 people.)

Job -------------------------------------------------------------------------
Open-ended Questions

In what year were you born?

Use open-ended when...

 Too many response categories (yr. of birth)

 you don’t wish to impose categories on respondent
 “really” consulting respondent
 qualitative--source of quotations
 determining appropriate categorization
 change-in-pace for respondent
Presence-absence questions

23. Have you ever had contact with handicapped in any of these
groups? (Circle to indicate "yes" or "no" for each group.)
Yes No
Community 1 0
Family 1 0
Relatives 1 0
Elementary school class 1 0
Junior high school class 1 0
Senior high school class 1 0
University class 1 0
As co-worker 1 0
Rank Ordering Questions

31. Rank order the three most important things you want in your job? (Place a 1 beside
the most important one; a 2 beside the next important one; and a 3 beside the next
most important one.)

High salary------------------- _____

Satisfaction------------------ ____

Continued interest----------- _____

Power------------------------- _____

Prestige---------------------- _____

Excitement-------------------- _____
Likert Style Questions

In the following items, circle a number to indicate the extent

to which you agree or disagree with each statement.

52. I would quit my present job if I won $1,000,000 through

a lottery.
Strongly Disagree 1 2 3 4 5 6 7 8 9 Strongly Agree
Tips for Likert Style Questions

 “and” alert: avoid multidimentional

 “strongly agree” always on right side
 avoid double negatives--use direct negative statements
 vary “strength of wording” to produce variation in response
 make all respondents feel comfortable
Types of questions—closed ended
 A. strongly agree, B. agree, C disagree, D strongly disagree.
 Check income category. Check years of education. Check age
category.
 Easy to enter data.
 Easy to compare answers.
Types of questions—open ended

 Participants own voice

 Doesn’t impose researchers categorization
 Confounded with language/verbal skills
 Often skipped or trite answers—How is your marriage?
Answer—OK, I guess.
 Structured questionnaires don’t have follow-ups to draw out
meaningful responses
 Difficult to analyze
Vague/Double Meaning
 Good questions involve one point or dimension.
 Real bad: Has your spouse yelled at you or hit you in the last
month?
 Problematic: How happy is your marriage?
– Happy regarding parenting
– Miserable regarding sexual relationship
– Great regarding equitable division of chores
– Terrible in terms of career goals
– Fantastic in terms of companionship
 Be as specific and unidimensional as possible
Avoid value laden wording

 Do you support the Tax Reform solution that will benefit so

many people including those on a fixed income, the elderly,
children, and the middle class?
Mitigate threatening questions

 Have you smoked cigarettes in the last month?

 Some people smoke. Did you happen to smoke in the last 12
months?
AVOID

 Avoid ambiguity, confusion, and vagueness.

– Make sure it is absolutely clear what you are asking and how you want
it answered.

– Avoid indefinite words or response categories.

AVOID
 Avoid emotional language, prestige bias and leading questions
– Watch out for loaded words that have a history of being attached to extreme situations.

– Watch for prestige markers that cue the respondent to give the "right" answer.

– Avoid leading questions.

– Avoid loading questions with extra adjectives and adverbs.

AVOID

 Avoid double-barreled questions

– Make each question about one and only one topic.

 Don't assume the respondent is an expert on

themselves (unless you have no choice)
AVOID

 Avoid asking questions beyond a respondent's capabilities

– People have cognitive limitations, especially when it comes to memory
of past events.

– It is pointless to ask people about things that are not natural ways for
them to think.
AVOID

 Avoid false premises

 Avoid asking about future intentions (if you can)

 Avoid negatives and especially double negatives

Formats: Grid

Response set—Reverse order--Format is efficient.

Item SA A D SD DK

1. I often feel depressed

    
2. I often feel I have no control
    
3. I control my own future
    
4. My boss appreciates the work I do
    
5. It is often difficult for me to sleep
    
Skip directions for self administered
Teenagers
1. Are you male or female?
 Male (go to question 4)
 Female (go to question 2)
2. Have you menstruated?
 Yes (go to question 3)
 No (go to question 4)
3. How much pain, if any, do you have when you menstruate the first time?
 None
 A little
 Quite a bit
 Extreme
4. How many days of school did you miss in the last month?
____ Enter the number of days
 Pretest
 Introduce--use blurb to introduce survey and to assure
confidentiality and anonymity
 ease them into it
 key and repeated questions at 1/3 point
 group questions by type--don’t jerk around
 anticipate computer data entry