Research Methods: Designing and Writing Effective Surveys

Research Methods
Designing and Writing Effective

Surveys
Overview
 Types of survey questions

 Common survey pitfalls
 Tips for improving your survey writing
 9 criteria for a good survey
Survey
 Survey method studies a sample of
individual units from a population.
Survey
 It involves asking questions from
respondents.
Survey
 It involves asking questions from
respondents.
 Divided into two broad areas:
Questionnaires and Interviews.
Types of Survey Questions
 Open-ended or Verbal
 The expected response is a word,
phrase, or an extended comment
 Responses can produce useful
information but analysis can present
problems
problems
 Some form of content analysis may be
required
problems
 Some form of content analysis may be
required
 Use few or stick with interviews
 Closed-ended, Likert Scale
Not very important Very Important
1 2 3 4 5 6 7
 The respondents must indicate by
selecting the number that best
represents their attitude
 Closed-ended, Likert Scale
Not very important Very Important
1 2 3 4 5 6 7
 The respondents must indicate by
selecting the number that best
represents their attitude
 A quick and easy way to measure more
different ranges of opinions, attitudes
 Easy to do analysis
 Closed-ended, Multiple Choice
 Which learning activity did you like best?
 A. Small group
 B. Role play
 C. Mapping
 D. Computer modeling
 Closed-ended, Multiple Choice
 Which learning activity did you like best?
 A. Small group
 B. Role play
 C. Mapping
 D. Computer modeling
When you want respondents to pick

the best answer or answers,
consider this type
Be sure to include specific directions
 Closed-ended, Ordinal
 Please write a number between 1 and 5 next to each item below.
Put a 1 next to the item that is most important to you in selecting
an on-line university course. Put a 5 next to the item that is
LEAST important. Please use each number only once.
 ___ availability of instructor for assistance
 ___ tuition cost for the course
 ___ ability to work in groups with other students
 ___ quality and quantity of instructor feedback
 ___ number of students enrolled
 Closed-ended, Ordinal
 Please write a number between 1 and 5 next to each item below.
Put a 1 next to the item that is most important to you in selecting
an on-line university course. Put a 5 next to the item that is
LEAST important. Please use each number only once.
 ___ availability of instructor for assistance
 ___ tuition cost for the course
 ___ ability to work in groups with other students
 ___ quality and quantity of instructor feedback
 ___ number of students enrolled
Great for rating things in relation to

other things
Again, be specific with instructions
 Closed-ended, Categorical
 How many hours do you study on a school night?
 0-1 2-3 3+
 Closed-ended, Categorical
 How many hours do you study on a school night?
 0-1 2-3 3+
Be sure that categories do not

overlap
Be sure to break down items into
distinct and clear categories
Each respondent must “belong” in a
specific category
 Closed-ended, Numerical
 How many years have you taught
full time?
 Closed-ended, Numerical
 How many years have you taught
full time?
 When the answer must be a real
number, ask a numerical question
Which One Do I Use?
 Use Open-ended questions…when you want to get
the respondent’s own words
 Use Likert-Scale questions to assess a person’s
feelings about something
 Use Multiple-Choice questions when there are a
finite number of options
 Use Ordinal questions to rate things in relation to
other things
 Use Categorical questions when the respondent
must fall into one section
 Use Numerical questions for real numbers like
age, number of months, etc.
Filtering/ Screening/Contingency
Questions
 Typically used when you want to filter out certain
respondents
 Respondents are directed to other questions in
the survey
 Too many jumps can confuse the reader
 Example:
 Have you ever visited a public hospital?
 Yes
 No
If “yes,” go to question #4
Common Survey Pitfalls: Bad
Questions Stink!
 Double barreled questions
 Use of complex and technical language
 Halo effect
 Social desirability
 Double negatives
Barreling vs. Specificity
 Barreled questions ask respondents
to rate 2 or more behaviors or
issues in a single question
 It is impossible to attribute one
answer to either issue
 This is the most common problem
with survey questions
Complexity vs. Simplicity
 Complexity occurs when sentence
phrasing is so long and labored that
respondents become confused
 Keep questions short and focused
on single issues
Let’s correct the complexity…
 Bad example: “What is the frequency of
the overall interpersonal, informal, and
formal communication between the focal
group and the targeted semi-
autonomous, functionally specialized
groups?”
 How can we improve this question?
 Good example: “What is the frequency of
the formal communication between the
focal group and your group?”
Social Desirability vs. Realism
 This occurs when questions have an
obviously “correct” or socially desirable
answer
 When respondents worry about giving the
“right” answer, the data will be distorted
 Make sure the question prompts
respondents to answer with a reasoned
opinion instead of an emotional response
Halo Effect vs. Respondent Autonomy
 Watch out for questions that link a

position with a particular person or
group
 The respondent’s attitude about the
person or group may influence their
attitude about the position
 Eliminate the link to a specific
person or group by creating an
unnamed group or dropping the link
Let’s correct the Halo effect
 Bad example: “Do you agree with
President Bush that schools need to
be accountable to taxpayers?”
 How can we improve this question?
 Good example: “Do you believe
that schools need to be accountable
to taxpayers?”
13 Tips for Improving Survey Writing
 1. Remember your survey’s purpose

 All other rules and guidelines are based
on this one
 There is a reason you decided to spend
time and effort doing a survey and you
should ensure that every question you
ask supports that reason
 If you start to get lost while writing your
questions, refer back to this rule
 2. If in doubt, throw it out

 This is another way of stating the first
rule
 A question should never be included in a
survey just because you can’t think of a
good reason to discard it
 If you cannot come up with a concrete
research benefit that will result from the
question, don’t use it
 3. Keep your questions simple
 Compound sentences force respondents
to keep a lot of information in their heads,
and is likely to produce unpredictable
results
 The following question is too complex for
a clear, usable answer and needs to be
broken down into component parts
 Example: Imagine a situation where your department chair is out sick,
two new students have enrolled mid-term, and the district’s test scores are
about to be released next week. How supported do you feel by the
administration?
 4. Stay focused—avoid vague issues

 If you ask “please rate your satisfaction
with the school’s discipline policy” the
answers will not lead to any specific
action steps
 Particular elements of the school’s
discipline policy must be probed if
responses are to result in specific
recommendations
 5. Include only one topic per question

 “Please rate your satisfaction with the
implementation and enforcement of your
school’s dress code” combines two issues
 You need to break this question into two
smaller ones in order to get responses
that can lead to recommended actions
 6. Avoid leading questions
 It is easy, and incorrect, to write a
question that the respondent believes has
a “right” answer
 “Scientists and doctors believe that
exercise is good for you. Do you agree?”
is an example of a leading question
 Even the most well-meaning researcher
can bias results by including extra
information in a question
 7. Consider alternative ways to ask sensitive
questions
 Income, drug or alcohol consumption, religious
beliefs, and political views are obvious examples
of sensitive topics
 But even topics like teaching styles, identifying
biases (racial and gender), classroom
management enforcement, and coworker
relations can be sensitive
 Questions like “did you vote in the last election”
forces respondents into a corner—they might be
unwilling to admit they did not vote because of
civic pride or embarrassment
 8. Make sure the respondent has enough
information
 Asking respondents “How has the school board
affected your classroom” isn’t as effective as “Last
month the school board passed a ruling that
requires a district committee composed of parents
and teachers to review all textbooks. Did you
know this?” followed by “What kind of changes to
your department’s curriculum have you seen?”
 Break these kind of questions into two parts- a
screening item and a follow-up question
 9. Response questions need to be mutually
exclusive and exhaustive
 If response questions are not mutually exclusive,
the respondent will have more than one
legitimate place for their answer
 The response choices “1-2,” “2-3,” and “more
than 3” pose a problem for someone who answers
“2”
 You must also ensure that the response options
you provide cover every possibility
 Providing a list of “reading,” “math,” and
“science” for “What is your toughest class?” isn’t
exhaustive enough
 10. Keep open-ended questions to a
minimum
 While open-ended questions are a
valuable tool, they should not be
overused
 They can result in respondent fatigue
where you’ll only get short answers
 These short answers can be avoided by
using a set of well-designed, closed-
ended questions
 Open-ended questions also pose problems
in terms of coding and analysis
 11. People interpret things differently,
particularly when it comes to time
 Trouble spots include “always,”
“sometimes,” and “never”
 You must build in a time frame of
reference to ensure that all respondents
are answering in the same way
 Example: “I am going to give you a list of
magazines. For each one, please indicate
whether you have read it regularly. By
regularly I mean at least twice a week.”
 12. Consider a “don’t know” response
 It is useful to allow people to say they simply do
not have an opinion about a topic
 Some researchers worry that people will opt for
that choice, reducing the ability to analyze
responses
 Evidence shows this fear is largely unfounded
 If you only want information from those with an
informed opinion about an issue or interest in a
topic, offer a “don’t know” choice
 13. Provide a meaningful scale
 The end points of Likert scales must be anchored
with meaningful labels (e.g. strongly agree, strongly
disagree)
 An odd number of points provides a middle
alternative and provides a good way for
respondents to head for the center
 If measuring highly variable opinions, use a scale
with a greater number of points
 You generally gain nothing by having a scale with
more than 7 points
Conclusion: 9 Criteria for a Good
Survey
 1. The target population (the group of
people from whom you want feedback) is
well-defined
 2. The actual people you survey matches
your target population
 3. The group you survey is large enough
to gain useful data
 4. Good follow-up minimizes non-
response
 5. The type of survey used is appropriate
Conclusion: 9 Criteria for a Good
Survey
 6. The questions are well-worded

 7. The survey is properly timed,
both in the amount of time it takes
to complete it, as well as when it is
distributed
 8. The people giving the survey are
well-trained
 9. The survey answers your
research questions

Research Methods: Designing and Writing Effective Surveys

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Research Methods: Designing and Writing Effective Surveys

Uploaded by

Copyright:

Available Formats

Research Methods

Designing and Writing Effective

 Types of survey questions

When you want respondents to pick

Great for rating things in relation to

Be sure that categories do not

 Watch out for questions that link a

 1. Remember your survey’s purpose

 2. If in doubt, throw it out

 4. Stay focused—avoid vague issues

 5. Include only one topic per question

 6. The questions are well-worded

You might also like