Example: A Question of Roles: Pursuing The Meaning

Chapter 9 Designing questionnaires
BOX 9.6
Example: a question of roles

Q. 2 For each of these jobs, please say whether you think it is appropriate for men only, for women only
or appropriate for both men and women.
Appropriate Appropriate for Appropriate for Don’t know

for men only women only both women
and men
Childminder 1 2 3 8
Firefighter 1 2 3 8
Primary school teacher 1 2 3 8
Midwife 1 2 3 8
Soldier in ‘front line’ action 1 2 3 8
Staying at home to look 1 2 3 8
after the children
Priest or minister 1 2 3 8
Secretary 1 2 3 8
Source: The Life and Times Survey 2000. Used with permission.
Pursuing the meaning

The more structured the enquiry (and the more structured the data collection instrument),
the more important it is to be rigorous in pursuit of the meanings we attach to the things we
are measuring. In a structured (quantitative) project we design a set of questions that cannot
be easily modified in the course of data collection. With a less structured (qualitative) project
we may start off with several sets of meanings or dimensions of a concept. The purpose of the
research may be to understand the meanings that the respondents place on these, or it may be
to refine and define these further, either as an end in itself or for feeding into the next stage of
a more structured piece of research. Whatever the purpose, we must start off with clear defini-
tions of the concepts that we are measuring before we can formulate the questions, otherwise
the data we get from the questions might be ambiguous at best and meaningless at worst.
Now that we know what we want to measure we have to think about how best to word the
questions. We need to turn the concepts and variables we identified into meaningful, objec-
tive questions that measure what we want them to measure. In addition, we want to design
Copyright © 2013. Pearson Education, Limited. All rights reserved.
questions that the respondent is willing and able to answer.
Question wording
What you are trying to achieve in wording a question is to ensure that you get valid and reli-
able data. To this end, each question should be worded so that the following hold:
● it measures what it claims to measure;
● it is relevant and meaningful to the respondent;
● it is acceptable to the respondent;
283
McGivern, Yvonne. The Practice of Market Research EBook : An Introduction, Pearson Education, Limited, 2013. ProQuest Ebook Central,
M09_MCGI3115_01_SE_C09.indd 283 4/22/13 12:55 PM
http://ebookcentral.proquest.com/lib/immgsm/detail.action?docID=5173519.
Created from immgsm on 2021-04-26 09:52:37.
Part 2 Getting started
● the respondent (and the interviewer) understand(s) it;

● it is interpreted in the way in which you intended;
● it is interpreted in the same way by all respondents;
● it elicits an accurate and meaningful response;
● the meaning of the response is clear and unambiguous.
Achieving all of this is far more difficult than it might at first appear, even for seemingly
simple, straightforward questions, as the two examples in Box 9.7 show.
BOX 9.7
Example: what are you asking me? question like this (from the 2010 Life and Times
Survey):
Put yourself in the respondent’s place. On first hear-
ing or reading the two questions below you might Q. 32 What is your personal income before tax
think that they are fairly straightforward (if some- and national insurance contributions? Please just
what intrusive in the case of the first one). But as you give me the number on the card.
start to think about your answer you might wonder, SHOW CARD
‘What exactly are you asking me?’ [Income bands plus ‘I do not wish to answer
this question’. The interviewer can code ‘Don’t
Q. How much money do you earn? know’ but this response is not on the show card.]
INCLUDE ALL INCOME FROM EMPLOYMENT
● What do they mean by ‘earn’? Money earned AND BENEFITS
in employment or money earned on investments
or from social benefits or a total amount earned
regardless of the source? What if I’m not work-
Q. Do you have a personal computer?
ing, say I’m retired or unemployed. Does this ● What do they mean by ‘personal computer’? A
mean I have no ‘earnings’? computer that I personally own? Or are they refer-
● To what time period does this apply? Do they ring to a type of computer, for example a desktop
want to know how much I earn in a year, a computer or a laptop or a hand-held computer?
month, a week? Do they want to know my earn- ● What do they mean by ‘you’? Me personally, or
ings in the last calendar year, the last financial or the household or family unit in which I live or the
tax year or the year up to the date of the inter- organisation for which I work?
view? Do they want to know earnings before or
● What do they mean by ‘have’? Do they want to
after tax or other deductions?
know whether I own a PC or have access to one

If it is personal income that you want to or the use of one? Do they mean at work or at
find out about, you might consider asking the home?
Writing effective questions

The examples in Box 9.7 highlight some of the problems that can arise when questions are
vaguely worded and not specific enough. In seeing the responses to such questions, would you
be confident in knowing what it was you had measured? In hearing or reading a question, the
respondent must be able to understand precisely what it is you are asking about. So, what do
you need to bear in mind in order to write effective questions? Below is a guide to the basics.
Use simple, straightforward grammar – active rather than passive voice, simple rather than
complex sentences. Avoid convoluted questions, e.g. Have you personally, in the last months,
284
M09_MCGI3115_01_SE_C09.indd 284 4/22/13 12:55 PM
travelled abroad on holiday (not including visits to friends and relatives) for a stay of four
days or more? Avoid negatively phrased questions, e.g. public speeches against racism should
not be allowed. Do you agree or disagree? Do you agree that it is not the job of the govern-
ment to take decisions about the following? Avoid double-barrelled questions, that is, ask-
ing two questions in one, e.g. Do you like using email and the web? Did you find the article
interesting and informative?
Keep questions (and statements and definitions) as short as possible – Oppenheim (2000)
suggests 20 words; if a question or set of questions requires an introduction, for example, at the
beginning of a new module or section of the questionnaire, aim to keep it to about 30 words.
Use simple, everyday words, e.g. live rather than reside, start rather than initiate, shop
rather than retail outlet. Avoid jargon and technical language unless it is suitable for and
understood uniformly by your entire target audience. Avoid abbreviations – not everyone
in your target audience may be familiar with them. Avoid using words or phrases that are
difficult to pronounce or read out, e.g. ‘In an anonymous form’. Use precise, specific rather
than general or abstract terms. Where appropriate, illustrate what you mean with an exam-
ple and/or provide clear and precise definitions of concepts or terms you have used in your
question (see the examples in Box 9.8 below).
Specify a reference period – taking account of the degree of precision required in relation
to the research objectives, the type of usage or behaviour you are asking about and what the
respondent can be reasonably expected to remember. For example, you might want to ask how
often respondents visit the cinema. You could ask this in a number of ways. Aim, however, to
avoid using adverbs such as ‘regularly’ or ‘frequently’ as these can mean different things to dif-
ferent respondents. It is preferable to give respondents more precise quantifiers, for example,
‘never’, ‘almost never’ or ‘several times a week’, or ‘about once a week’ and so on. You might even
want to know the actual number of visits they make on average in a month; or the actual number
of visits they made last month. For questions about use or behaviour that occurs often, a shorter
reference period is usually more suitable; for use or behaviour that happens less frequently,
a longer reference period is more appropriate. In asking about usage or behaviour during a
particular time period, a week for example, you need to decide whether it is appropriate to ask
about ‘in the last week’ or ‘last week’, or ‘last week, that is, the seven days ending last Sunday’.
BOX 9.8
Example: definitions and reference periods Q. Apart from special occasions such as weddings,
funerals, baptisms and so on, how often nowadays
Q. 17 Please think back over the last 12 months do you attend services or meetings connected with
about how your health has been. Compared to peo- your religion?
ple of your own age, would you say that your health
has on the whole been. . .
Q. Do you have a long-standing illness, disability or
(Please tick one box only ❏)
infirmity? By long-standing I mean anything that has
excellent ❏ troubled you over a period of time or that is likely to
good ❏
fair ❏ affect you over a period of time?
poor ❏
or very poor ❏
(can’t choose) ❏ Q. Some people have extra family responsibili-
Source: The Life and Times Survey 2010. Used with permission.
ties because they look after someone who’s sick,
➨
285
M09_MCGI3115_01_SE_C09.indd 285 4/22/13 12:55 PM
Box 9.8 continued
disabled or elderly. May I check, is there anyone liv- a sick or disabled or elderly relative, wife, husband,
ing with you who is sick, disabled or elderly whom partner, child, friend)?
you look after or give special help to (for example, Source: The Life and Times Survey and the National Centre for
Social Research. Used with permission.
Avoid asking hypothetical questions about respondents’ likely future behaviour – you run
the risk of getting meaningless, hypothetical data. Instead, give as much detail and context as
you can within the question. If you include material that allows respondents to think them-
selves into the situation you are asking about – you can do this using vignettes or scenarios
– then you are more likely to get meaningful data. Some examples of questions that ask about
hypothetical situations are given in Box 9.9 below.
BOX 9.9
Example: Vignette or scenario style Q. What, if anything, would you do? CODE ALL
questions THAT APPLY
Q. I am now going to ask about a few hypothetical

Yes No
situations. I would like to remind you that there are Complain to member of staff 1 2
no right or wrong answers, only opinions. immediately
You are queuing up at a small post office. It is Leave/go to another Post 1 2
near closing time or last post collection. A person Office
with a speech impairment is taking a long time to be Leave/come back another day 1 2
served as they are speaking very slowly. How would Politely ask another member 1 2
you feel? SHOW CARD. CODE ALL THAT APPLY of staff if they could open
another window to serve
Yes No Nothing, keep waiting 1 2
Annoyed, irritated 1 2 Ask Post Office staff if I could 1 2
Threatened, frightened 1 2 help out
Suspicious, dubious 1 2 Ask person being served if I 1 2
Nervous, anxious 1 2
could help out
Uncomfortable, embarrassed 1 2
Surprised, shocked 1 2 Other 1 2
Curious 1 2 Don’t know 1 2
Worried of adverse reaction/offending 1 2
person
Sorry for the person 1 2 Source: The Life and Times Survey 2003. Used with permission.
Understanding, genuinely concerned 1 2
Fine, would not bother me 1 2
Other 1 2
None of these 1 2
Don’t know 1 2
Another pitfall is embedding assumptions in the question wording, for example, ‘How
often do you travel to France?’, and the classic, ‘When did you stop beating your wife?’ Try
to answer these questions and you will see where the problem lies. You must also be careful
286
M09_MCGI3115_01_SE_C09.indd 286 4/22/13 12:55 PM
v that you do not word a question in such a way as to lead the respondent towards a particular
answer, for example, ‘Do you always in fact buy the most expensive brand?’, ‘To what extent
do you agree that the service is meeting your needs?’, and ‘Do you agree that it is right that
your organisation makes donations to political party X?’ You can also be guilty of leading
respondents by using loaded, ‘non-neutral’ words, for example, ‘What do you think of welfare
for the poor?’ Remember rule B14 of the MRS Code of Conduct which seeks to ensure that
questions do not lead respondents to a particular point of view.
You need to take as much care in wording responses to questions as you do in wording
the questions themselves. We look in more detail at question responses later in the chapter.
Finally, in writing the question, make sure it – and any definition or description of the
topic, concept or terms used, and the instructions to the respondent and/or the interviewer
– are complete. Neither the respondent nor the interviewer should have to rely on his or her
own interpretation, or, in the case of the interviewer, have to use his or her own words to
explain any part of the question.
Next, we look at questions that deal with what can broadly be called ‘sensitive’ topics and
at the questioning techniques for dealing with them.
Questions on sensitive topics

If not handled properly – clearly worded, in the right place on the questionnaire, the ques-
tion and the answer recorded without embarrassment on the part of the interviewer –
questions on sensitive or embarrassing topics can lead to refusals – refusals to answer the
question or to continue with the interview, or refusal to take part in the first place. What is
judged to be intrusive, embarrassing or sensitive varies enormously; and what is a straight-
forward issue to the researcher may be a particularly sensitive issue to the respondent, and
vice versa (Lee, 1992). Subjects that tend to be sensitive to most people and in most cultures
include money, voting, religion, sexual activity, criminal behaviour, and use of alcohol and
drugs. One way of handling responses to sensitive questions in a face-to-face interview is
to ask respondents to fill in the answers on the questionnaire themselves on the screen or
on a separate self-completion sheet as in Case study 9.5. Alternatively, show cards, from
which the respondent reads out a code for his or her response, can be used, as in Case study
9.3. The relative anonymity of a telephone interview or an online survey often makes these
approaches unnecessary.
Social desirability bias

Questions on some topics are susceptible to a form of response bias known as social desirabil-
ity or prestige bias. Here responses do not accurately gather respondents’ actual behaviour,
attitudes or opinions, for several reasons. For instance, a respondent might decide to choose
the response they think is ‘socially desirable’, one that is viewed more favourably by society
rather than the one that actually applies to them. According to Sudman and Bradburn (1983),
over-reporting occurs in questions about being a good citizen; being a well-informed and
cultured person; and fulfilling moral and social responsibilities (for example, in questions
about completing accurate tax returns, driving to the speed limit, using your vote, frequency
of visiting museums and art galleries and going to the theatre, giving to charity and recycling
waste). Prestige bias can also affect answers to questions about age, occupation, income,
and cleanliness and grooming. Under-reporting occurs in relation to issues such as alcohol
consumption, sexual activity, criminal activity and use of illegal drugs. Another source of
287
M09_MCGI3115_01_SE_C09.indd 287 4/22/13 12:55 PM
social desirability bias arises when a respondent does not want to appear uninformed about
the topic under question and so rather than reply, ‘don’t know’ or ‘not sure’, he or she gives
a response. A further source of social desirability bias relates to questions on sensitive or
embarrassing topics such as illness.
In designing questions to avoid this type of bias you need to make it just as easy and pain-
less for the respondent to give the low prestige answer as it is to give the high prestige answer.
This can be done in the same way as questions about sensitive topics – via a self-completion
questionnaire, or using show cards or shuffle packs from which the respondent reads the
relevant code. Another way is to use indirect questioning (e.g. ‘What do you think people
would think. . . ’. ‘Other people have told us. . . ’), or to ensure that the question is presented
in such a way that all answers are allowable and equally acceptable, or to offer the respond-
ent a valid escape route. Some examples are given in Box 9.10 (see Brace and Nancarrow,
2008, for further examples and how they fare in practice). As with sensitive topics, the more
anonymous methods of data collection – telephone and self-completion (postal or online) –
may be better suited to collecting this type of information.
BOX 9.10
Example: techniques to overcome social desirability bias

Q. 1 Talking to people about the general election on. . . , we have found that a lot of people
didn’t manage to vote. How about you – did you manage to vote in the general election?
IF NOT ELIGIBLE/TOO YOUNG TO VOTE: CODE ‘NO’.
Yes, voted 1 Ask Q. 1a

No 2 Go to Q. 2
Source: British Social Attitudes Survey. Used with permission.
Q. 19 Some people say that even if they feel they are prejudiced, they try to overcome
their feelings and avoid displaying prejudiced behaviour. What about you? Would you say
that . . . READ OUT
You avoid displaying prejudiced behaviour towards minority ethnic people 1

Your behaviour towards minority ethnic people is consistent with the prejudice you feel 2
(Don’t know) 8
Source: The Life and Times Survey, 2010. Used with permission.
Q. 32 What is your personal income before tax and national insurance contributions? Please
just give me the number on the card.
SHOW CARD [Income bands plus ‘I do not wish to answer this question’. The interviewer
can code ‘Don’t know’ but this response is not on the show card.]
INCLUDE ALL INCOME FROM EMPLOYMENT AND BENEFITS
Source: The Life and Times Survey. Used with permission.
288
M09_MCGI3115_01_SE_C09.indd 288 4/22/13 12:55 PM
Case study 9.3
How do you ask that?

This case study highlights the questionnaire design There are, of course, problems with all questionnaires
issues that arose in researching AIDS and shows how but they applied a fortiori in this case. In tackling
they were successfully resolved. Some interviews took them we built on our experience and the accumulated
place in gay bars (see Case study 7.1) and some took wisdom of survey research. Little in our approach
place in the respondent’s home. was entirely new but we do believe it represents an
impressive and valuable case history in collecting
Why this case study is worth reading sensitive data.
This case study is worth reading for many reasons: it The three basic tools, which we subsequently dis-
shows questionnaire design in action – it shows what cuss, were:
was done and why; it illustrates some key techniques ● Confidentiality
and gives a rationale for their use; it discusses the
● A progression of hurdles in the interview which,
impact of questionnaire design and interviewing tech-
when cleared, moved the respondent on to increas-
nique on data quality (and on the interviewer and
ingly more sensitive or personal subjects
respondent experience of taking part in research).
The key words are: length, respondent fatigue, sen- ● The use of coded shuffle packs and prompt aids for
sitive, embarrassing, inhibited, interviewing environ- the vast bulk of the questions.
ment, confidentiality, progression, hurdles, coded
shuffle packs, quality control, structure of question- Confidentiality
naire, random order, code checking, literacy, grid, Interviewing in gay bars we take no names and
self-completion booklets, quality of data, wording of addresses (see Case study 7.1). For the in-home inter-
questionnaire, instructions, honesty, valid data. view of the general public we assure respondents of
confidentiality and say that there is no need to take
Introduction names and addresses. However, at the end of the inter-
Questionnaire design for this project faced the ‘normal’ view we ask whether the respondent would be will-
problems of balancing comprehensiveness against length ing to be re-interviewed on a similar subject. About
and respondent fatigue. It also faced the much more dif- 80 per cent agree and in these cases it is ‘natural’ to
ficult task of eliciting highly sensitive information from take names and addresses, but they are written on
potentially inhibited respondents. For purposes of com- different forms and posted to us separately from the
parison it was highly desirable that the same question- interviews. These are used to conduct our normal
naire could be used for both the general public and gay quality-control backchecking procedures. Letters are
samples. Although it was anticipated from qualitative left with respondents as a reassurance at the end of
work reported elsewhere (Siddall et al., 1987) that the the interview. Whilst these procedures are valuable for
latter would be relatively uninhibited in discussing sex- the respondent, we also believe they are valuable for
ual matters, in February 1986 no such assumption could the interviewer in reassuring her of the confidentiality
be made about the general public. There were also the and contributing to a confident approach to her task.
problems of inhibition in relation to other people present In all cases, both at bars and in homes, interviewers are
during the interview, or to the interviewer herself. instructed to try to obtain the interviews with respond-
Thus the key issues were: ents on their own. This happens in about 60 per cent of
interviews with the general public and 75 per cent of
● How to accommodate the very different interview-
interviews in gay bars.
ing environments (‘gay bar’ v ‘Englishman’s castle’).
● How to minimise inhibitions vis-à-vis eavesdroppers Clearing hurdles
(either a friend in the bar or a wife or husband at The general structure of the questionnaire is one of pro-
home). gressive hurdles, moving to ever more sensitive subjects,
● How to minimise inhibitions vis-à-vis the inter- starting with fairly innocuous questions about general
viewer and enable the respondent to give honest health hazards and ending up with detailed questions
answers to very personal and sensitive questions. on sexual behaviour in quite explicit terms – for example
➨
289
M09_MCGI3115_01_SE_C09.indd 289 4/22/13 12:55 PM
Case study 9.3 (continued)
whether they have engaged in anal intercourse, with or ● They give respondents time to read and consider
without a sheath, in the last three months. each statement and when they see the pattern of
The actual coverage of the questionnaire is as their answers to make minor adjustments.
follows. ● They are much faster in administration.
Spontaneous beliefs about health hazards. ● Respondents are more involved and less likely to
become fatigued.
*Awareness of AIDS in context of seven other
diseases. ● They are less tiring for interviewers who also have
time for code checking and thought-gathering.
*Rating of eight diseases for how common or rare
they are, how serious for people who catch them,
In the present case they have one further overwhelming
degree of respondent knowledge, extent of threat to
advantage – they enable the communication of sensi-
health of nation, personal concern about catching.
tive information on potentially very embarrassing sub-
*Spontaneous and prompted knowledge of ways of jects, without either the respondent or the interviewer
catching AIDS, and the people at risk, and ways of having to say any words that might inhibit them.
reducing risk. We realised it would be critical to train respondents.
*Attitudes to AIDS. This is done at the very early stages. They are asked to
Ease of obtaining information about AIDS. sort a set of eight cards each with a name of a disease
and a number printed on it into those they have heard
*Reactions if relative caught AIDS.
of and those they have not. The interviewer then asks
Source of information about AIDS. for the numbers only. Moreover her questionnaire only
Advertising recall. has numbers on it, so even if names are given she has
*Attitudes to advertising. to ask for the numbers before she can code. (The code
that she has to ring is the same as the number on the
Awareness of and reaction to leaflets.
card which greatly speeds up the process.) The cards
*Attitudes to homosexuality. for the diseases the respondent has heard of are then
Whether know homosexual. shuffled on three occasions. After each shuffle he has
Blood donation and reasons for lapsing. to sort them into categories on a different board to
show respectively how common they are, how serious
Media exposure. it would be for someone who caught them and how
Demographic and other classification. much he feels he knows about each. After this series
*Heterosexual behaviour questions. of questions the respondent understands what to do
with each subsequent shuffle pack as it is handed to
*Men only: sexual orientation and homosexual
him. He gets on with the task and knows that at the end
behaviour questions.
he will be required to call out the numbers in response
Willingness to be re-interviewed. category order.
The question areas marked* are administered by shuf- Having carried out a large number of interviews
fle packs (or self-completion) which is our most impor- ourselves, we can vouch for the fact that at the end the
tant tool. interviewer has virtually no idea of the respondent’s
views on anything – it is genuinely confidential. More
over, when anyone else is present they too are only
Shuffle packs and booklets
hearing an exchange of numbers. In addition, the pro-
BMRB interviewers are familiar with the use of shuf-
cess of training the respondent helps rapport and inter-
fle packs and sorting boards in conventional surveys.
action with the interviewer.
We believe they have many advantages over the more
The system speeds up the process of coding while
frequently used methods of using a prompt card with a
allowing the respondent time to consider each answer.
scale on it and the interviewer reading out the attitude
There are on occasion difficulties with literacy of some
battery or self-completion scales. For example:
respondents, but these are far outweighed in our view
● They enable the concepts to be presented to each by the absence of respondents mishearing attitude
respondent in a unique and random order. statements. Indeed we do not believe a method of
➨
290
M09_MCGI3115_01_SE_C09.indd 290 4/22/13 12:55 PM
Case study 9.3 (continued)
verbal administration of attitude batteries would have monogamous people about their sexual activity with
been possible in the disco type conditions in many of their only partner).
the gay bars . . .
Altogether seven question areas with 118 pieces of Quality of the data
information are covered by different coloured shuffle Although we have taken very great care with the
packs and similarly coloured sorting boards, compared approach and wording of our questionnaire, it is diffi-
with only 30 questions (plus classification) obtained by cult to be sure whether people end up telling the truth
standard question and answer methods, with or with- or not (as it is in all survey research). At the end of
out prompt cards. These techniques take us through the interview the interviewers are asked to assess the
most of the questionnaire and classification leaving respondent’s honesty. Typically, 75 per cent of the gen-
only the questions on sexual behaviour. eral public and about 85 per cent of the gay sample are
For those who qualify we then have a self-comple- coded as ‘very honest’ and virtually everyone else as
tion booklet which asks in very considerable detail ‘quite honest’. We also ask respondents how worthwhile
about sexual behaviour in the last three months and they feel it has been to give their time to being inter-
twelve months – numbers of partners, types of sexual viewed. About 80 per cent of the general public and
activity indulged in, sex with bisexuals, prostitutes, over 90 per cent of gays say that it has been ‘extremely’
drug injectors and so on. These booklets are handed or ‘very’ worthwhile.
over to respondents together with an envelope into
which they are placed after completion, sealed, and
handed back to the interviewer. References
Because of the known outspokenness of the gay Siddall, J., Stride, C. and Sargent, J. (1987) ‘Are you
community from the qualitative work it was felt that homosexual, heterosexual or bisexual? If so, you could
detailed sexual behaviour would probably be obtain- develop AIDS’, Proceedings of the Market Research Soci-
able from all the gay sample. However, for the gen- ety Conference, London: MRS.
eral public we restrict the booklet to those who claim Source: Adapted from Orton, S. and Samuels, J. (1997) ‘What we
to have had two or more heterosexual partners in the have learnt from researching AIDS’, International Journal of Market
last year (we thus avoid, for example, asking older Research, 39, 1, pp. 175–200, www.ijmr.com.
As we noted above, it is important to maintain the respondent’s interest throughout the

questionnaire. The design of individual questions and the questionnaire as a whole has a
role to play in this. Repetition, in the form of banks of similar questions, lists of tick or click
box answers, and so on, can, at best, bore respondents, leading them to answer automati-
cally, with little thought or involvement and, at worst, lose them altogether. Either way, you
end up with poor quality data. Another way to lose respondents is to ask them irrelevant or
unnecessary questions. Every question should be relevant to the research objectives (if not, it
should be removed) and relevant to the respondent’s situation or experience, and on a subject
that he or she can reasonably be expected to answer accurately. If a question is irrelevant to
a particular subset of respondents then routeing instructions should be used to ensure that
they are not asked the question.
The context for online questionnaires – the online environment – offers huge scope for
designing effective, engaging questionnaires. Think of some of the most engaging websites,
apps or online games you know. Yet few online surveys are as well designed to engage you.
In Box 9.11 are suggestions offered by a practitioner in this area, Jon Puleston (2011), to
improve online surveys. He proposes that we can learn from methods used in qualitative
research, social psychology and game playing.
291
M09_MCGI3115_01_SE_C09.indd 291 4/22/13 12:55 PM
BOX 9.11
How to make the most of an online survey

Here Jon Puleston of Global Market InSite Inc (GMI) this can help stimulate twice as much feedback. Ask-
Interactive summarises the main findings of research he ing respondents if they want to do a voluntary part
and colleagues (Puleston, 2009 and 2010; Puleston and of a survey can be an extremely powerful technique,
Sleep, 2008; and Puleston, Brechin and Mintel, 2011) too – rather than reducing the volume of feedback it
conducted into how online survey completion rates and can result in an increase in the time and care spent
data quality could be improved. on the answer.
● Learning from qualitative researchers. The way they
● Start by looking at a survey as a form of creative
ask questions is very different from the language
communication to a large audience.Surveys are
used in many online surveys. We found that pro-
commonly delivered in 12-point plain text, with little
jection methodologies and tasking exercises, like
or no imagery or animation, and with respondents
mood-board building, work brilliantly. Respond-
asked to digest 20 or 30 options on a page. Apply-
ents tend to enjoy performing more intellectually
ing a few basic design considerations can have a
involving tasks, and placing questions into a more
dramatic effect. In experiments, we reduced dropout
imaginary framework stimulates their imaginations
by up to 75 per cent simply by making surveys more
and can lead to much richer data. For example, we
visually attractive.
asked respondents to evaluate a new product idea
● Engage respondents from the beginning.Any good as if they were a judge on a game show called ‘New
qualitative researcher knows how important it is Product Factors’, and this trebled the volume of
to ‘warm up’ focus group participants, and the insightful feedback.
same applies to online survey respondents. These
● The value of piloting. Piloting an online survey is
are often expected to begin a survey by reading a
quick and easy, and can be extremely valuable, but
large block of text, but our experiments found that
less than five per cent of research projects we run
less than 50 per cent do so properly. Breaking the
are piloted to test the effectiveness of the ques-
information into sound-bites, telling a story, adding
tionnaire. The benefits of piloting online are clear.
some imagery and humour, results in respondents
Observing how questions are answered by a test
investing more time in the survey and giving more
sample of 50 respondents eliminates the guesswork
thoughtful feedback.
in judging how the overall sample will do so. Mis-
● Adoption of more creative questioning methods. takes can be spotted and corrected, and the fram-
Avoiding lines of tick boxes can reduce straight- ing of questions and options refined, eliminating
lining and neutral scoring, and radically improve redundancy and increasing quality of feedback. We
respondents’ enjoyment. piloted a global ethnographic research study for
● Understanding the critical role of imagery. Images Sony Music. We conducted seven waves of pilots,
not only increase respondent engagement, they also the questionnaire design was gradually improved,
activate memory and imagination, and are crucial along with the use of imagery and engagement
when asking people to evaluate brands and per- techniques outlined above, which resulted in an
sonalities. An image can be used to generate more increase in respondent feedback from 130 words
ideas, and, used carefully throughout a survey, can to over 400.
guide respondents through the experience and ● The power of game-play. In the future, we believe
increase response rates. Imagine if TV only had text! that successful survey design will incorporate many
● Learning from social psychology. Error messages techniques developed through an understanding of
and forced conditionality of open-ended responses game-play, understanding why people treat some
are often very counter-productive in online surveys tasks as games, into which they will pour time and
– they annoy respondents. Far better to use social effort for little or no reward, and some as work,
compliance techniques, like showing them a good which they might perform resentfully even when
example of what someone else has written, to estab- rewarded. Completing most online surveys is cur-
lish a social benchmark that they copy. We found rently seen as work, with the reduction in willingness
➨
292
M09_MCGI3115_01_SE_C09.indd 292 4/22/13 12:55 PM

Example: A Question of Roles: Pursuing The Meaning

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Example: A Question of Roles: Pursuing The Meaning

Uploaded by

Copyright:

Available Formats

Chapter 9 Designing questionnaires

Example: a question of roles

Appropriate Appropriate for Appropriate for Don’t know

Pursuing the meaning

questions that the respondent is willing and able to answer.

● the respondent (and the interviewer) understand(s) it;

know whether I own a PC or have access to one

Writing effective questions

Q. I am now going to ask about a few hypothetical

Questions on sensitive topics

Social desirability bias

Example: techniques to overcome social desirability bias

Yes, voted 1 Ask Q. 1a

Source: British Social Attitudes Survey. Used with permission.

You avoid displaying prejudiced behaviour towards minority ethnic people 1

Case study 9.3

How do you ask that?

Case study 9.3 (continued)

Case study 9.3 (continued)

As we noted above, it is important to maintain the respondent’s interest throughout the

How to make the most of an online survey

You might also like