Lesson Designing, Testing, and

1 Revising Survey
Questionnaires

What’s In

Before we start exploring the topic, let us find out if you can distinguish a
well-written survey question from a confusing one. Write YES if the question is
good, and NO if otherwise.

1.How short was Napoleon?

2.How would you describe Napoleon’s height?

3.Do you think special car seats should be required for infant passengers?
4.Should concerned parents use infant car seats?
5.How satisfied or dissatisfied are you with the pay and work benefits of your

current job?

Notes to the Teacher

This module prepares the learners for them to be able to
design, test, and revise survey questionnaires within the
standards of writing academic papers.

What’s New

Surveys are an effective way to collect data from your customers. They are helpful
in evaluating your offerings and finding ways to make changes for the better. Yet,
some people struggle with survey design and creation and create surveys that
aren’t giving them the most accurate results.
Answer the sample survey below then assess it using the rubric below. Write your
score in every criterion in your answer sheet. If the total score is 28, then it is a
good survey questionnaire.

What is It

How to Write a Reaction Paper?

•
 Layout The selection of The selection of The selection of The selection
graphics, line styles, graphics, line styles graphics, line styles of graphics,
and arrangement and arrangement and arrangement line styles
options enhances the options mostly options sometimes and
layout and meaning enhance the layout enhance the layout arrangement
of the survey. of the survey. of the survey. options do
not enhance
the layout of
the survey.

Content All essential Most of the essential Some of the essential One or fewer
questions are questions are questions are essential
properly addressed. properly addressed. properly addressed. questions are
addressed.

Spelling/ All words are spelled Most words are Most words are Numerous
correctly. Grammar, spelled spelled spelling
Grammar punctuation, spacing correctly. Grammar, correctly. Grammar, errors.
and word usage are punctuation, spacing punctuation, spacing Grammar,
appropriate. and word usage are and word usage have punctuation,
mostly appropriate. some errors. spacing and
word usage
have a
number of
errors.

Utility Easy to use and Easy to follow. Choppy, but gets the Difficult to
pleasant to look at. job done. follow and is
jumbled.

Total Score

What is It

Questionnaire: Definition, Examples, Design and Types

By Saul McLeod, updated 2018

A questionnaire is a research instrument consisting of a series of questions for the

purpose of gathering information from respondents. Questionnaires can be thought
of as a kind of written interview. They can be carried out face to face, by telephone,
computer or post.

Questionnaires provide a relatively cheap, quick and efficient way of obtaining large
amounts of information from a large sample of people. Data can be collected
relatively quickly because the researcher would not need to be present when the
questionnaires were completed. This is useful for large populations when interviews
would be impractical. However, a problem with questionnaires is that respondents
may lie due to social desirability. Most people want to present a positive image of
themselves and so may lie or bend the truth to look good, e.g., learners would
exaggerate revision duration.

Questionnaires can be an effective means of measuring the behavior, attitudes,

preferences, opinions and intentions of relatively large numbers of subjects more
cheaply and quickly than other methods. An important distinction is between open-
ended and closed questions. Often a questionnaire uses both open and closed
questions to collect data. This is beneficial as it means both quantitative and
qualitative data can be obtained.

Closed Questions

Closed questions structure the answer by only allowing responses which fit into
pre-decided categories. Data that can be placed into a category is called nominal
data. The category can be restricted to as few as two options, i.e., dichotomous (e.g.
“yes” or “no” “male” or “female”) or include quite complex lists of alternatives from
which the respondent can choose (e.g., polytomous).

Closed questions can also provide ordinal data (which can be ranked). This often
involves using a continuous rating scale to measure the strength of attitudes or
emotions. For example, strongly agree / agree / neutral / disagree / strongly
disagree / unable to answer. Closed questions have been used to research type A
personality (e.g., Friedman & Rosenman, 1974), and also to assess life events
which may cause stress (Holmes & Rahe, 1967), and attachment (Fraley, Waller, &
Brennan, 2000).

Strengths
• They can be economical. This means they can provide large amounts of
research data for relatively low costs. Therefore, a large sample size can be
obtained which should be representative of the population, which a
researcher can then generalize from.
• The respondent provides information which can be easily converted into
quantitative data (e.g., count the number of 'yes' or 'no' answers), allowing
statistical analysis of the responses.
• The questions are standardized. All respondents are asked exactly the same
questions in the same order. This means a questionnaire can be replicated
easily to check for reliability. Therefore, a second researcher can use the
questionnaire to check that the results are consistent.

Limitations
• They lack detail. Because the responses are fixed, there is less scope for
respondents to supply answers which reflect their true feelings on a topic.

Open Questions

Open questions allow people to express what they think in their own words. Open-
ended questions enable the respondent to answer in as much detail as they like in
their own words. For example: “Can you tell me how happy you feel right now?” If
you want to gather more in-depth answers from your respondents, then open
questions will work better. These give no pre-set answer options and instead allow
the respondents to put down exactly what they like in their own words.
Open questions are often used for complex questions that cannot be answered in a
few simple categories but require more detail and discussion.
Lawrence Kohlberg presented his participants with moral dilemmas. One of the
most famous concerns a character called Heinz who is faced with the choice
between watching his wife die of cancer or stealing the only drug that could help
her.
Participants were asked whether Heinz should steal the drug or not and, more
importantly, for their reasons why upholding or breaking the law is right.

Strengths
• Rich qualitative data is obtained as open questions allow the respondent to
elaborate on their answer. This means the research can find out why a person
holds a certain attitude.

Limitations
• Time-consuming to collect the data. It takes longer for the respondent to
complete open questions. This is a problem as a smaller sample size may be
obtained.
• Time-consuming to analyze the data. It takes longer for the researcher to
analyze qualitative data as they have to read the answers and try to put
them into categories by coding, which is often subjective and difficult.
However, Smith (1992) has devoted an entire book to the issues of thematic
content analysis the includes 14 different scoring systems for open-ended
questions.
• Not suitable for less educated respondents as open questions require
superior writing skills and a better ability to express one's feelings verbally.

Questionnaire Design

With some questionnaires suffering from a response rate as low as 5%, it is

essential that a questionnaire is well designed. There are a number of important
factors in questionnaire design.

Aims
Make sure that all questions asked address the aims of the research. However, use
only one feature of the construct you are investigating in per item.
Length
The longer the questionnaire, the less likely people will complete it. Questions
should be short, clear, and be to the point; any unnecessary questions/items
should be omitted.
Pilot Study
Run a small scale practice study to ensure people understand the questions. People
will also be able to give detailed honest feedback on the questionnaire design.
Question Order
Questions should progress logically from the least sensitive to the most sensitive,
from the factual and behavioral to the cognitive, and from the more general to the
more specific.
The researcher should ensure that the answer to a question is not influenced by
previous questions.
Terminology
There should be a minimum of technical jargon. Questions should be simple, to the
point and easy to understand.
The language of a questionnaire should be appropriate to the vocabulary of the
group of people being studied. Use statements which are interpreted in the same
way by members of different subpopulations of the population of interest.
For example, the researcher must change the language of questions to match the
social background of respondents' age / educational level / social class / ethnicity
etc.
Presentation
Make sure it looks professional, include clear and concise instructions. If sent
through the post make sure the envelope does not signify ‘junk mail.’
Ethical Issues

The researcher must ensure that the information provided by the respondent is
kept confidential, e.g., name, address, etc. This means questionnaires are good for
researching sensitive topics as respondents will be more honest when they cannot
be identified. Keeping the questionnaire confidential should also reduce the
likelihood of any psychological harm, such as embarrassment. Participants must
provide informed consent prior to completing the questionnaire, and must be aware
that they have the right to withdraw their information at any time during the
survey/ study.

Problems with Postal Questionnaires

The data might not be valid (i.e., truthful) as we can never be sure that the right
person actually completed the postal questionnaire.
Also, postal questionnaires may not be representative of the population they are
studying?
• This is because some questionnaires may be lost in the post reducing the
sample size.
• The questionnaire may be completed by someone who is not a member of the
research population.
• Those with strong views on the questionnaire’s subject are more likely to
complete it than those with no interest in it.
•

Benefits of a Pilot Study

A pilot study is a practice / small-scale study conducted before the main study. It
allows the researcher to try out the study with a few participants so that
adjustments can be made before the main study, so saving time and money. It is
important to conduct a questionnaire pilot study for the following reasons:

• Check that respondents understand the terminology used in the

questionnaire.
• Check that emotive questions have not been used as they make people
defensive and could invalidate their answers.
• Check that leading questions have not been used as they could bias the
respondent's answer.
Ensure the questionnaire can be completed in an appropriate time frame.
 DESIGNING A SURVEY QUESTIONNAIRE
Questionnaire research design proceeds in an orderly and specific manner. Each
item in the flow chart depends upon the successful completion of all the previous
items. Therefore, it is important not to skip a single step. Notice that there are two
feedback loops in the flow chart to allow revisions to the process and questionnaire.

Questionnaire Research Flow Chart

Design Methodology

Determine Feasibility

Develop Instruments

Select Sample

Conduct Pilot Test

Revise Instruments

Conduct Research

Analyze Data

Prepare Report

Establishing Goals. The first step in any survey is deciding what you want to learn.
The goals of the project determine whom you will survey and what you will ask
them. If your goals are unclear, the results will probably be unclear.

Selecting Your Sample. There are two main components in determining whom you
will interview. The first is deciding what kind of people to interview. Researchers
often call this group the target population. If you are trying to determine the likely
success of a product, the target population may be less obvious. Correctly
determining the target population is critical. If you do not interview the right kinds
of people, you will not successfully meet your goals. The next thing to decide is
how many people you need to interview. Statisticians know that a small,
representative sample will reflect the group from which it is drawn. The larger the
sample, the more precisely it reflects the target group.

Avoiding a Biased Sample. A biased sample will produce biased results. Totally
excluding all bias is almost impossible; however, if you recognize bias exists, you
can intuitively discount some of the answers Interviewing Methods.

Once you have decided on your sample you must decide on your method of data
collection. Each method has advantages and disadvantages.
Personal Interviews. An interview is called personal when the Interviewer asks the
questions face-to-face with the Interviewee. Personal interviews can take place in
the home, at a shopping mall, on the street, outside a movie theatre or polling
place, and so on.

Telephone Surveys. Surveying by telephone is the most popular interviewing

method in the USA. This is made possible by nearly universal coverage (96% of
homes have a telephone).

Computer Direct Interviews. These are interviews in which the Interviewees enter
their own answers directly into a computer Email Surveys.

Email surveys are both very economical and very fast. More people have email than
have full Internet access. This makes email a better choice than a Web page survey
for some populations. Email surveys are limited to simple questionnaires.

Internet/Intranet (Web Page) Surveys. Web surveys are rapidly gaining popularity.
They have major speed, cost, and flexibility advantages, but also significant
sampling limitations. These limitations make software selection especially
important and restrict the groups you can study using this technique.

Questionnaire Design
General Considerations

1. The first rule is to design the questionnaire to fit the medium. People
responding to mail or Web surveys cannot easily ask “What exactly do you mean by
that?” if they do not understand a question. Personal questions are sometimes best
handled by mail or computer, where anonymity is most assured.

2. KISS - keep it short and simple.

3. Start with an introduction or welcome message. A good introduction or

welcome message will encourage people to complete your questionnaire. When
practical, state who you are and why you want the information in the survey.

4. Allow a “Don't Know” or “Not Applicable” response to all questions, except to

those in which you are certain that all respondents will have a clear answer

Researchers use three basic types of questions: multiple choice, numeric open
end and text open end. Examples of each kind of question follow: