Guidelines for creating Questionnaires

Introduction
It is virtually impossible to give a set of rigid rules for setting up a
questionnaire. This document therefore provides only general guidelines.
Planning prior to setting up a questionnaire

The purpose of the investigation or research project must be established

before the questionnaire is drawn up. The purpose dictates what type of
information should be obtained with the questionnaire. At this stage the
researcher must do a thorough literature study on the subject under
investigation.
Decide whether a questionnaire is the most suitable method to capture
the desired information.
The purpose of the investigation will determine the target group, that is
the respondents to whom the questionnaire is directed.
The target group will determine the manner in which the questionnaire
is to be completed. In other words whether the questionnaires will be
posted to the respondents, completed in a group under supervision, be
completed during personal interviews, etc.
Consider the conditions under which the questionnaire is to be
completed. The conditions in a hut in the informal sector differ
considerably from conditions in an air-conditioned hall with comfortable
desks.
Discuss the size of the sample, the definition of the population and the
actual drawing of the sample with a statistician. The final draft of the
questionnaire must also be discussed with the statistician and the
research consultant. The hypotheses to be tested must preferably be
formulated beforehand to ensure that all relevant information is included
in the questionnaire.
Ensure that all information which may later be necessary for evaluation
of the realisation of the sample is included in the questionnaire.
The question of anonymity must be addressed. It must be clearly
explained to the respondent that all information will be considered
confidential and that no personal information will be published. No
information that can identify the respondent should be on the
questionnaire. For example, the questionnaires should not be numbered
prior to sending it to the respondent. An exception must be made in the
case of studies where information must be obtained before and after
certain events or actions and the two sets of data must be linked. In this
case information such as a student number must be used and the
situation must be explained to the respondent.
Do a pilot study to test the questionnaire. If this is not possible the

questionnaire must be tested on a few persons similar to persons in the

target group. This will help to eliminate possible inherent problems.

General remarks regarding the contents

The target group determines the level of the questions. The contents of
the questions as well as the language and the terminology should not
overestimate nor underestimate the abilities of the respondent.
The length of the questionnaire is determined by the size of the sample
as well as by the target group. A very long questionnaire frustrates and
exhausts the respondent. If the sample size is not sufficiently large,
questions with too many categories as possible answers can lead to
frequencies that are so low and "sparse" that the information can either
not be used at all or the categories must be grouped together to be of
any use at all.
The language must be correct and unambiguous. It is important that
the possible answers can be linked to the question. Questions must be
restricted to one topic only.
Psychometric questionnaires, that is questionnaires measuring aptitude,
IQ etc are not discussed here. Certain psychometric concepts are,
however, relevant to the development of questionnaires.
Construct validity indicates whether relevant constructs or concepts are
being investigated. For example, in a questionnaire about agricultural
problems, valid constructs would be droughts, maintenance of
implements, etc
Content validity concerns the contents of the questions measuring the
constructs. If the construct is, for example, maintenance of implements
then "quantity of lubricants bought" is valid, but "amount spent on
dipping fluid" is not.
Reliability indicates the extent to which the questionnaire will give the
same results repeatedly. Say, for example, the question is "How much
did it rain?" The possible answers "Plenty" and "Not much" will not have
the same meaning in wet years as in dry years. On the other hand
"More than 40mm", "30-40mm" and "Less than 30mm" will have a
constant meaning.

A few concepts

Variables e.g. Gender, home language, denomination, distance

Associated with each variable is one of the following:
Categories e.g. Male, Female, Afrikaans, English, Tswana
Values e.g. 10, 15, 20km or 5, 10, 15 years
Scale values e.g. Excellent, Good, Moderate, Poor
Scale types
Nominal e.g. Male, Female
Ordinal e.g. Excellent, Good, Moderate, Poor or the five classes

of a Likert Scale
Interval/Ratio e.g. Temperature, Mass, Distance, Age in year
intervals

Classification of question types

Questions can be classified according to their function, which in turn is
determined by the purpose of the questionnaire. The classification in the
following table is a guideline. It is, however, important to remember that it
seldom happens that only one kind of question appears in a questionnaire.
Purpose

Type of question

Description

Biographic

Explaining

Behaviour and knowledge

Planning

Knowledge

Predicting

Attitude

Discussion of different types of questions

Biographical questions

Only the necessary information must be collected.

Questions requiring sensitive information from the respondent must be
carefully worded.
Age must preferably be posed as an open question, and if necessary,
categorised at a later stage. This enables one to use a larger variety of
statistical methods.
The statistical techniques that will be used must be kept in mind during the
compilation of the questions. If the researcher plans to use techniques such
as regression analysis and analysis of variance then the information must,
where possible, be collected in an uncategorised form.
Questions must, as said before, be unambiguous and not open to
misunderstanding.
Poor example:
Do you travel by bus in the mornings or in the evenings?
Better option:
Do you travel by bus in the mornings?
Do you travel by bus in the evenings?

Yes/No
Yes/No

Yes/No

The possible answers provided for a question must be logical and relevant
for the target group. Once again, the researcher must be sure that the
required information is collected.
For example, if the target group is children between 13 and 16 years of age
and the question is about highest educational qualification, the following
options will not give much information
None / Grade 3-7 / Grade 8-12 / B-degree / Post-graduate
Instead, the following options would be more relevant
None / Grade 3-6 / Grade 7 / Grade 8 / Grade 9 / Grade 10 / Grade 11 /
Grade 12
Questions designed to determine frequency of events can be problematic.
Once again, the solution is a consistent clear formulation.
Poor example:
How often do you consult a doctor?
Regularly, frequently, occasionally, never
The concept frequently will definitely not have the same meaning for
everybody
Better example:
How many times did you consult a doctor between January, 1 2003 and
December,31 2003?
Once, two or three times, four or five times, more then five times, never
In the last example the period is clearly defined, the frequency of visits is
stated unambiguously and provision is made for respondents who did not
visit the doctor.
Sifting questions
In the previous example a sifting question can be used and followed by an
open question.
Example:
Did you consult a doctor during the period.........?
Yes/No
If yes, how many times?
........
The advantage, again, is that more statistical techniques can be used.
Open questions
Open questions have no formulated categories for possible answers. This
type of questions are usually used when insufficient knowledge regarding
the particular subject exists and the researcher is uncertain whether
predefined categories will cover all possibilities. Open questions are,
therefore, particularly useful for exploratory studies. It can also be used to
obtain more information regarding previous closed questions or to test the

intensity of the response. As the coding of this type of question is time

consuming it should only be used when absolutely necessary.

"Answer each one" or "Choose only one"

The use of these two types of answers depends on the question and the
purpose of the study. "Answer each one" is used when there is more than
one possible answer to the question. It also forces the respondent to take
each given category into consideration. The following method is
recommended:
Why did you start smoking?
Wanted to know what it felt like

Yes

No

I was dared

Yes

No

I wanted to prove myself

Yes

No

I did not want to be different from my friends

Yes

No

Boredom

Yes

No

I heard that it was relaxing

Yes

No

If the respondent was asked to choose all relevant reasons, as in the next
example, it cannot be assumed that a reason that has not been chosen
yields "No" as in the previous example.
Why did you start smoking? Choose all relevant reasons.
Wanted to know what it felt like
I was dared
I wanted to prove myself
I did not want to be different from my
friends
Boredom
I heard that it was relaxing
"Choose only one" is used in cases where the question indicates that the
most important reason or aspect must be chosen.
The target group must be considered, as it is sometimes easier for
respondents to answer Yes/No to each possibility than to indicate the most
important reason.
Ranking
Questions that expect the respondent to rank a number of reasons, causes
or anything else must only be used when the researcher is sure that the
respondents will be able to manage this. Even then the number of
possibilities to be ranked must be kept to a minimum. It must also be
clearly indicated that the respondent must rank every possibility.

Direct against sympathetic approach

Example:
Are you overweight?
Against:
Do you think that you are overweight?
Again there can be no fixed rule. The researcher must be guided by the
target group, the nature of the study and the specific question. Sometimes
a more sympathetic approach or a question that is worded in a
conversational style would be more effective in motivating a respondent to
answer a sensitive question. The length of the question must, however, be
restricted as much as possible.
Time dimension of events
Do not overestimate the respondent's memory. Questions must be
restricted to important or recent events.
Unreasonable demands must not be made on the respondent. The relevant
time period must be clearly defined.
Example:
Did you buy a new car during the past year?
What is regarded as the past year? 2003? April 2003 to March 2004?
It is sometimes possible, given the subject and the size of the sample, to
make deductions over time by restricting the question to a specific time.
Example:
How many times did you go to church during August?
(Where August is the month preceding the survey)
If the sample size is sufficient and one assumes that the church attendance
pattern is relatively constant, it is reasonable to expect that exceptional
behaviour will be cancelled out and that the monthly church attendance of
respondents will be reflected.
Reliability
It is difficult to check the answers to biographical questions and these
questions must be formulated in such a way that the respondent can give
honest answers.
Knowledge questions
Most questionnaires include such questions not to test the respondents
knowledge as such but because knowledge is the foundation of certain
behaviour or because it helps to form attitudes.
Sifting questions are valuable in this case, if a respondent indicates that
he/she has never heard of a chiropractor, then it does not make any sense
to ask this specific respondent any questions regarding the work of a
chiropractor.

10

The respondent's frequency of contact with the subject must also be taken
into account. The knowledge about bus routes of a person who travels by
bus every day will be more "reliable" than the knowledge of a person who
only travels by bus occasionally.
For the same reasons as discussed previously with the Yes/No answers, the
preferred type is True/False. In some cases "Don't know" or "Uncertain" can
be added as a category. The two possibilities are not equivalent and must
be handled carefully.
Questions regarding opinions and attitudes
With regard to attitudes, there is a difference between determining the
nature of the attitude and determining the intensity of the attitude.
In the first case the answer to each question is important and must be
reported separately. In the second case a number of questions that
theoretically measure the construct must be included. Techniques like
factor analysis must then be used to ascertain that this is proved in
practice. The group of questions that clearly measures a construct is then
used as a entity and values or totals can be reported for the constructs.
The different scales that can be used will now be discussed.

Dichotomous scale
Example:
The bus service satisfies my needs

Yes

No

Buses depart on time

Yes

No

Bus tariffs are too expensive

Yes

No

In this case the answer to each question is analysed on its own.

Likert scales
This type of scale can be used to measure intensity of feelings. As
mentioned before, a specific theoretical construct is supposed and with
the help of statistical techniques it must be determined whether the
data confirms the theory.
Example:
Strongl
y Agree

Agree

Neutral

Disagree

Strongly
Disagree

Travelling by bus
is time
consuming

Buses are always

late

Bus drivers drive

11

recklessly

12

Diagram scale
This scale determines the respondents' tendency to both sides of a
central point.
Example:
Punctuality
::

::

::

::

::

::

::

::
0

::

::

::

::
+

Semantic differential scale

This is an extension of the diagram scale as the concepts that are
measured in words to the scale.
Example:
BUS SERVICE
Excellent :_:_:_:_:_:_:_: Poor
Clean :_:_:_:_:_:_:_: Dirty
Quick :_:_:_:_:_:_:_: Slow
Whenever this scale is used it is important to give a proper example.

General remarks regarding scales

When deciding which statistical techniques to use it must be
remembered that the distances between the possible answers do not
invariably have the same value.
The number of categories that are used must be restricted to the
minimum.
A problem, that may arise with certain target groups, is that respondents
choose all the neutral answers or answers that are considered to be
"desirable". This problem can, largely be avoided by giving proper
instructions, asking relevant questions and, while coding the
questionnaires, being mindful of problem cases.

The difference between knowledge, opinions and attitudes

The difference is not always very clear and the literature sometimes
differentiates on a continuum between cognitive and evaluative attitudes.
In general, a question with "Don't know" as a possible answer indicates
measurement of knowledge, whereas a question with "Unsure" as a
possibility in turn indicates measurement of attitude.
The phrasing of questions
This is the most important aspect of the questionnaire and must be
properly attended to.
Example:

13

Speeches against democracy should not be allowed.

will elicit a different response to:
Speeches against democracy should be prohibited.
"should not be allowed" is more neutral than "should be prohibited"
Adjectives indicating intensity should be used with care.
Hypothetical questions rarely yield much information. The respondent must
be familiar with the subject or object regarding which his attitude is being
determined.
Where examples on how to answer questions are given, these must be
clearly separated from the rest of the question and identified as an
example. Examples must merely show how to answer questions and not be
indicative as to the type of answer that is expected.
Sequence of questions
As a general guideline similar questions and questions regarding a common
subject should as far as possible be grouped together to give the
questionnaire a logical flow. In some cases it might be better to put
sensitive questions, that the respondent may possibly refuse to answer, at
the end of the questionnaire. In this way the maximum amount of the
information will be gained from the respondent.
Source
Adapted and translated by Rina Owen from:
CS Steenekamp: "Praktiese riglyne vir vraelyskonstruksie"

14