You are on page 1of 49

Constructing questions for

interviews & questionnaires
Theory and practice in social research
William Foddy

Preface
• These slides include one theoretical
framework for constructing questions and a
small set of good and bad practices for
constructing questions
• This is not only or complete set, but gives
hints about what should you think when
constructing questions.

Errors in gathering data through
survey procedures are
• Responders’ failure to understand questions an intended
• A lack of effort or interest on the part of responders
• Respondents’ unwillingness to admit certain attitudes or
behaviours
• The failure of respondents’ memory or comprehension
process in the stressed condition of the interview
• Interviewers failures of various kinds (e.g. the tendency to
change wording, failures in presentation procedures and
the adoption of faulty recording process

Examples that illustrate the inadequate of
many of the questions
• Factual questions sometimes elicit invalid answer: As simple as age
• Relationship between what respondents say and what they actually do in not always
very strong: E.g. racism: Questions: Do you let a black person inside a bar? Do they
really?
• Responders’ attitudes, beliefs, opinions, habits interests often seem to be
extraordinary unstable
• Small changes in wording sometimes produce major changes in the distribution of
responses
• Respondents commonly misinterpret questions
• Answer to earlier question can affect to later question
• Order of options changes the respondents: e.g. middle option
• Format of question: Open vs. close
• Respondents answer even if they don't know about the topic
• Cultural context affects

The principal assumptions that have defined the
general orientation adopted by survey researcher in
the past









The researcher has clearly defined the topic about which information is
required
Respondents have the information that researcher requires
Respondents are able to access the required information under the conditions
of the research situation
Respondents can understand every question as intended
Respondents are willing (or at least, can be motivated) to give the information
Responses are more valid if they know why that is asked
Answers are more valid if they are not suggested
The research situation does not affect to the results
The process does not affect the respondent
All responses are meaningfully comparable

The traditional survey model
A carefully standardised physical
stimulus (i.e. question)

Researcher/
Interviewer

Respondent
A response (i.e. answer)
expressed in term of a
standardised format provided by
the researcher

The key issue: the comparability
of aswer
• The reseacher must be clear about the nature of the information
required and encode a request for this format
• The respondent must decode this request in the way the researcher
intends it to be decoded
• The respondent must encode an answer that contains the information
researcher has requested
• The reseacher must decode an answer as the respondent intendedd it to
be encoded

Symbolic interactioinist theory
• Human beings interpret and define each other's actions. Not only
simple stimulus-response
• Human beings can be objects of their own attention. Argue themselves,
take pride themselves…
• Conscious social behavior is intentional behavior. Situation affects.
• Interpreting, planning and acting are ongoing process which begin
anew at every stage of a social process
• In situation there is a role and person wants to hold on it.
• Always exists
• In brief: social situation is constantly negotiating a shared definition of
the situation; taking one another's viewpoint into account; and
interpreting another's behavior

A model of symbolic interactionist view of question
answer behaviour
Interviewer

Responder

Encodes question, taking into
account own purposes and
presumptions/knowledge about
the respondent, and perceptions
of the respondent's
presumptions/knowledge about
self

Decodes question, taking into
account own purposes and
presumptions/knowledge about
the interviewer, and perceptions
of the interviewer's
presumptions/knowledge about
self

Encodes question, taking into
account own purposes and
presumptions/knowledge about
the respondent, and perceptions
of the respondent's
presumptions/knowledge about
self

Encodes question, taking into
account own purposes and
presumptions/knowledge about
the interviewer, and perceptions
of the interviewer's
presumptions/knowledge about
self

The implication of the symbolic interaction
theory for social research
• Roles
• Respondents try to find a mutually shared
definition of the situation
• Interpretation of researcher acts
• Clues of what kind of information
researcher wants

Other
• Multidimensionality. Nothing is onedimensional
• Level of generality in the statements
• Level of generality in the topic
• Utterance frame: descriptive, explanatory or
evaluative

Defining topic properly
Interviewer

Responder

Encodes question.
The need to clearly specify the
information that is required. The
assumption that the respondent
have the required information.
The assumption that the
respondents can access the
information

Decodes answer

Decodes question

Encodes answer

Researcher has clearly defined
the required information
• Defined topic
– Global vs. local (29)
– Dimensions (economic, technical...)
• Is PL good thing: Marketing vs. Technical

– E.g. better social secure in cost of taxes

• Researcher has clear idea what kind of
information is required

Responder have required
information
• Hypothetical question => hypothetical Answer
• Willingness to answer vs. information to answer
• Opinions vs. facts
– Grounded opinions
– Serious opinions

• Responds are capable of verbalising information.
– Not hurry, give time to answer

Formulating intelligible requests
for information
Interviewer

Responder

Encodes question.
Formulating an intelligible
request for information.

Decodes question

Defining key terms for
respondents and virtues of
simplicity and breity

Decodes answer

Encodes answer

The meaning of individual words
• Context specific meanings
– Ambiguities words
– Different meanings
– Cultural decencies

• Relative difficulty of words
– foreign words

• The operation of unintended nuances associated
with apparently similar words
– Forbid vs. allow

The meaning of concepts
• Lack of empirical references
– How should be
– Difficul, unclear, wide concepts

• Context decencies (often, usually etc.)
• Concrete: Not: often, quite often, seldom.
Use 1,2,3,4,5

Structural complexities
• Number of words
• Less is better, but not complex ones
• Grammatical
– Asking too much in one question
– divide & conquer
– not what you like this and that!

• Not too much negatives: Double is bad

Good practices
• Descriptive introduction: I'd like to
describe…
• Explain before question
– The addition of clauses, phrases and
instructions

• Anecdotes
• Education of the respondent affects

Contextual influences on respondents'
interpretation of questions
Interviewer

Responder

Decodes question
Encodes question.

Contextual clues that influence
respondents’ interpretations

Decodes answer

Encodes answer

Clues afforded by either the
Question itself or its components
• Clues associated with the question
– Some existing idea always
– Leading questions

• Clues in components
– Additional phrases
– Note: Vegetables e.g. spinach not associated vegetarian
in general as intended

Clues in response options.
• Approve or disapprove - not only approve in question
• Scale
• Pre-set response options as memory clues



May not remember other options
Information range covered by the options.
Who counts own averages?
Abstract: no knowledge of normative levels

• Response biased with number of options
– Respondent favor first when reads, last when hears - vary the order
– Let responder tell, if no-one match

Impact of preceding questions
• Influence associated with prior Q. Issues:
– Does one have any knowledge?
– How deep is knowledge or how thorough
– How one interprets

• Psychological need for consistent
– Need to stay in one opinion

• Even-handedness
– Fair for all sides

Impact of preceding questions
(cont’d)
• The impact of the previous answers
• The impact of the overall situation
– Semantics of human mind.
– Willingness to give information

Contextual influences on respondents'
interpretation of questions
Interviewer

Encodes question.

Responder

Decodes question

Encodes answer
Decodes answer

The kind of answer given is defined by the
dimension of the response framework that
is employed

Descriptive accounts

How to describe the situation
Perspectives

Level of social generality of responses

You? Singular or plural?

Explanations
• Respondent can frame an explanation in many
different ways
• Why did you do X




Causal antecedent - what caused to do X
Goal antecedent - purpose for X
Enablement factors - how possible
Causal consequences - what happened after X
The researcher's expectations

Evaluations
• Always relative
• Standards necessary does not exists
• Evaluative standard that are external to the
question
– Pleasant/easy flight - pilot vs. passenger

• Evaluative standard that are built in to the question
– Agree -disagree

Limitations of human memory

Limitations of human memory
• Long term memory problems
– Nobody remembers all
– Ask from present to past, not past to present.

• Short term memory problems
– Simple questions

Filters

Filters
• Establishing the relevance of questions to
respondent
– Respondents tend to answer all the questions
• “I don't know enough”

– Don't know vs. don't have opinion
– Position of filters
• Middle and don’t know are problematic
• Middle category at all

– subjective viewpoint to strongly (dis)agree
• How strongly?

Reducing question threat

Reducing question threat
• Always some threat
• Biased
• Refuse

How?








Casual, Do you happen to have…?
Imputation of deviance, You know everyone does.
Anonymous
Lessen psychological immediacy of the Q (e.g. other instead self.
Numeric coding of alternatives)
Decrease specificity of the information called (e.g. broad response
categories)
Adopt knowing so that respondents have to confirm rather than volunteer
'Kinsey' straight at eyes and ask
Adopt indirect so that respond give answer without knowing
Place threatening Q at the end of series

How? (cont’d)
• Door in the face: Ask direct, if does not
answer ask indirect/about e.g. salary.
• Ask long, don’t hesitate top repeat, give
time and encourage to use time
• The definition of Q threat

Threat causes bias
• Topics that are desirable and over-reported
– Be good citizen
– Be well informed and cultural person
– Have fulfilled of moral and social responsibilities

• Topics that are socially undesirable and under-reported
– Illness and disabilities
– Illegal and contra-normative behavior
– Financial status

• Feeling of guilt or personal fears
• Threat associated with the nature of the relationship between interviewer and
respond
– Social equity
– Fear of political or economical sanctions

Open vs. closed question debate

Open vs. closed question debate
• Coding responses to open questions
• Formulating response option for closed
question

Evaluation of the use of open questions
• Open Q doesn’t suggest answer -- or does it?
– Respondent’s use of "probes"

• Indicate level of knowledge
– Assumptions
• will answer open Q if knows
• don't try to answer open Q if don't know
• wil answer closed question if don't know

• Answers indicate the salience of the topic in the responder's mind
– Is there evidence?

• Avoid format effect
– Not in the middle!
– Is it in the closed?

Evaluation of the use of open questions
• Allows complex motivation
– Indicates more how respondent has interpreted it
– Indicate motivation that have influenced respondent's orientation to the
topic
– Indicates the frame of reference

• Problems associated with probing inadequate answer
– Turns to close

• Problems associated with coding response!

Evaluation assumptions
associated with closed questions
• Answers the question in the same way
-meaningfully comparable?
• Easier to answer
• More easily analyzed
– Interpretations

• Problems associated with recording responses to
closed questions
– Start with general instructions
– Explain why to answer

Measuring attitudes

Measuring attitudes
• List of respond alternatives p. 153
• Define topic clearly
• Applicability of the topic to respondents has to be
established
– Don’t know

• Respondents has to know what sort of answers they should
know
– Level of generality

• Specs of standards
– "Strongly agree"

Measuring attitudes
• Stimulus centered effects
– Number of categories
• 7 +/- 2

– Anchoring effects of he category labels
• The word in the positive/negative end?

• Problems in the batteries of rating scales
– Meaning can alter
– Some category
• long list. Don't know last when answering the first

• Ambiguity

Checks to ensure that questions
work as intended

Checks to ensure that questions
work as intended
• Editing rules (list p. 184)
• Piloting Question
– Observation hard, but must be done in the beginning to
ensure that questions work e.g. had to repeat?

• Question testing
– Rephrase the question in responder's own words
– Double interview
• Come in the beginning

– Allow aloud thinking

Tieto vai mielipide
Halu vastata vai
Vihjaa että pitäisi olla? vastaus olemassa?


Mikä on tyypillinen
tuote?




Esim KKK

Koponentti parametrinen vs
komponetin mukanaolo parametrista
Mikä on komponentti?

Kuinka monta parametroitua komponenttia on
tyypillisessä toimitettavassa tuotteessa (0=ei käytetä)?
Miten paljon parametreja on tyypillisessä
parametroidussa komponentissa?
Parametrien määrän vaihteluväli?
Millaisia parametreja komponenteissa on?
Millaisia arvoalueita parametreilla on (kokonaisluku,
arvoalue, yksi kokonaisluku joukosta , yksi annetuista
vaihtoehdoista, reaalilukuja, joukkoja)? Mitä muita?

Jos ymmärsi aluksi väärin, halu
konsistenttiin ilmaisuun

Strukturaalinen kompleksisuus, liian
monimutkainen lause:
Kun toimitatte tuotteen, tuotteessa om
komponentteja. Kompoenenteilla voi
olla toimituskohtaisia parametreja.
Onko olemassa tällaisia
komponentteja, joissa on parametreja?
Montako tälläistä kompoenttia, jossa
on parametreja, tuotteessa on?