Cognitive Interviewing Techniques:

A Brief Overview1

Introduction

Cognitive interviewing was developed by survey methodologists and psychologists as a

means of evaluating and preventing sources of error in survey questionnaires. Cognitive,
or cognition, defined by Merriam-Webster, means “the act or process of knowing
including both awareness and judgment.”2 Cognitive interviewing focuses on drawing out
the cognitive processes of respondent as they answer survey questions. It is used to study
both the covert processes, that are usually hidden, and the overt or more readily
observable processes.

Cognitive Theory

The general model consists of the following different processes of respondents when
answering a survey question:

Comprehension of the question:

· Question intent: What does the respondent believe the question to be asking?
· Meaning of terms: What do specific words and phrases in the question mean to
the respondent?

Retrieval from memory of relevant information:

· Recallability of information: What information does the respondent need to

recall in order to answer the question?
· Recall strategy: What strategies does the respondent use to retrieve information
– Do they count each event individually or do they try to estimate?

Decision processes:

· Motivation: Does the respondent devote sufficient mental effort to answer the
question accurately and thoughtfully?
· Sensitivity/Social Desirability: Does the respondent want to tell the truth? Does
he/she feel pressure to answer a certain way?

Response processes:

· Mapping the response: Can the respondent match his or her internally generated
response to one of the response options given for the question?
1
Based on Cognitive Interviewing: A “How To” Guide developed by Gordon B. Willis at the Research
Triangle Institute.
2
www.m-w.com
Prepared by Kristen Cibelli of Benetech, Inc. for use in Sierra Leone, January 2004
Cognitive Interviewing Methods

There are two main types of cognitive interviewing methods: think-aloud and verbal
probing. Explanations of both methods are provided below.

1) Think-aloud

In the think-aloud method respondents are instructed to think aloud as they answer the
survey questions.

Advantages: freedom from interview-imposed bias, minimal interviewer training

requirements, open-ended format

Disadvantages: need for subject training, subject resistance, burden on the subject,
tendency for the subject to stray from the task, bias in subject information processing

2) Verbal Probing

In this context, probing refers to a process of investigation using questions. In the verbal
probing method the interviewer probes further into the basis of the response to a survey
question by following up with a series of related questions. (We will focus on this
method).

Categories of cognitive probes include:

· comprehension/interpretation probe
· paraphrasing
· confidence judgment
· recall probe
· specific probe
· general probes

Advantages: control of the interview, ease of training of the subject

Disadvantages: artificiality, potential for bias

There are two approaches to verbal probing:

· concurrent (more frequently used) and

· retrospective (useful when testing self-administered questionnaires and in later
stages of questionnaire development)

How are probes developed?

Prepared by Kristen Cibelli of Benetech, Inc. for use in Sierra Leone, January 2004
 · Scripted probes: A set of standard probes are prepared in advance – This is good
if there is adequate time to prepare and for less experienced interviewers and
provides more standardization.

· Spontaneous probes: Each individual interviewer creates probes on the spot –

This can be considered haphazard but often the best probes come up in the
process of the interview and cannot always be planned ahead of time - a unique
relationship developed in each interview between the subject, interviewer and
questionnaire.

· Combination: The interviewer uses a combination of standard probes as well as

probes he or she develops on the spot.

Cognitive interview examples

The following is a list of potential problems that can be identified through a cognitive
interviewing process:

· Comprehension problem: The respondent does not understand what the question
is asking.
· Multiple interpretations: The respondent identifies alternate, but equally
reasonable, interpretations of the question.
· Difficulty to recall: The respondent finds it difficult to recall with the degree of
precision necessary.
· Wrong type of response: The questions fails to draw out the type of data the
questionnaire designer intended.
· Social pressure: The respondent feels pressure to respond in a certain way (not
honestly).
· Format of the question: Format of the question is tedious to the respondent –
questions with multiple response options for example. Try to come up with a
better way of asking the question.
· No response: The respondent does not have an answer to the question -
sometimes people just don't have a response to a question - if this is the case,
revising it won't help - it can be better to delete the question.

Detecting Structural (logical) Problems in Survey Questions

Apart from the individual questions on the survey, it is important to test the overall
questionnaire for structural or logical problems.

· Learning about the topic from people who are experts or have first-hand
experience - such as drug-users for a survey on drug use.

· Test the survey with people for flaws in the logic, layout, making sure that the
questions and/or the response options apply to all respondents (For example:

Prepared by Kristen Cibelli of Benetech, Inc. for use in Sierra Leone, January 2004
 How long have you owned your house? - What about renters?)

Using Cognitive Interviewing in Sierra Leone

Cognitive interviewing is a technique that has been used in the United States, with a great
deal of success, to improve questionnaire design. Here in Sierra Leone, we will be doing
cognitive interviewing in six different languages – Krio, Mende, Temne, Kono, Koranko,
and Limba. We will interview approximately four people in each of these languages.
Part of our task will be to compare what we find across all of the languages and determine
edits to the survey that will correct the problems that have occurred in specific languages
but still maintain consistency of the questions across the languages. In other words, every
conceptual edit must be replicated in each language. This is a monumental task that will
require we communicate regularly about what you are finding. Our first meeting will be
at 5pm tonight.

You will be performing the cognitive interviewing in teams. One person will have the
role of asking the questions and probing, and the other person will have the role of taking
notes on what the respondent says, including the respondent’s tone of voice and body
language. We want to understand which questions make the respondent the most and
least comfortable, as well as how well the respondent understands the question. Today
each team will be monitored by either Kristen or Jana, and they may ask for you to
translate responses or suggest probing questions to use.

Our biggest concern is getting respondents to tell us years in which particular events
happened. We will need to develop a set of probes we can use in the field to elicit the
years in which certain events happened. Possible probes include:

· How many rice harvests have passed since that happened?

· How many years before the end of the war did that happen?
· How old was that person when that happened? (Especially useful if a birth date is
known).

The remainder of our training will involve two tasks. First, Jana will demonstrate
cognitive interviewing by interviewing one of you in English. Then we will discuss ways
in which we will elicit date information from respondents.

Prepared by Kristen Cibelli of Benetech, Inc. for use in Sierra Leone, January 2004