Cognitive interviews:

Training for the European Web

Survey on Drugs
Kateřina Škařupová, João Matias, Susana Mota
EMCDDA
26 and 29 June 2020
Welcome & objectives

• International projects: IPA7 (& EU4MD)

• 2016: Validity and reliability of the EWSD CZ_Q
• Lessons learned: EWSD 1+2 & Mini-EWSD on
COVID-19

Objectives:
• refresh the basics
• provide guidance for the EWSD questionnaire testing and
reporting
• highlight some aspects of the EWSD questionnaire
• motivate you to invest time and resources in testing the Q
Outline and practicalities

• EWSD process, timeline, and questionnaire

• Cognitive interviews: theory and objectives
• How to: techniques and practical considerations
• 20’ Q&A

During the training:

• Stay muted to avoid echo.
• Keep questions for Q&A, or write them in the chat

3
EWSD process flow & timeline

Training
LimeSurvey
(June 25,30)

Training Decisions based

cognitive testing Finish draft on testing taken,
(June 26,29) questionnaire final questionnaire

June 2020 July 2020? EMCDDA to create Enter and test February/March
Training online
master online 2021
recruitment (Sep
questionnaire questionnaire
28,30)
online (Oct) (NFPs) Oct- Jan

Translation/testing of the questionnaire (July to September)

Develop recruitment plan - NGOs, media, ads (September til launch)

 EWSD Questionnaire: Modules and Logic

A bit about you Cannabis module Socio-demographics

• Age, gender, country • Herbal and resin • Soc-dem

• Warm-up questions • Patterns of use • Referral
• Drug use (LMP/LYP/LTP) • Purchasing practices • COVID-19

• Specific drug modules only shown to

those that used the drug in the past 12
months

• If respondent used more than one drug:

• Drug modules displayed in random
order

• After each drug module, respondent

can chose to continue with another
drug, or proceed to final module
New in 2021
• COVID-19
• Low-THC/CBD
• Reduced NPS
5
What are cognitive interviews?
What is cognitive interview

• In-depth, semi-structured administrations of

the survey questionnaire designed to reveal
problems and insufficiencies of the survey
tool

• Different from survey administration

• Purpose and method
• Specific interviewer’s skills
• Different materials (online/paper)

8
Objectives of cognitive interviews

• Reduce error
• Assess construct validity
• Minimize bias and assess cross-group/cultural
invariance
• Reduce respondent’s burden and improve the
experience
• To understand better the data –
ethnographer’s approach

9
What type of problems can cognitive interviews
reveal?

• Insufficient, overlooked, misinterpreted instructions

• Misunderstood wording

• Vague or ambiguous definitions/terms

• Items not relevant to respondent

• Confusing response options, or response formats

• Illogical structure: order, routing

• Contextual problems

11
Silent misunderstandings

• When respondents answer with ease, but

interpret questions differently than intended

MTF: Have you ever used Ecstasy / Molly?

12
Known issues in the EWSD questionnaire

13
Substance names & slang

Cannabis
Herbal - weed/skunk
Resin - hasish?

Cocaine powder vs. crack-cocaine

Base/smokable?

Ecstasy/MDMA

Any other drug.

14
NPS definition

New substances that imitate the effects of illicit

drugs (such as cannabis, ecstasy, cocaine, etc)
may now be sometimes available. They are
sometimes called (insert ‘local name’ such as,
‘new psychoactive substances’ ‘legal highs’,
‘research chemicals’) and can come in different
form, for example – herbal mixtures, powders,
crystals or tablets. When did you use any of these
substances?

15
Low-THC & CBD products

• New items to be tested.

• How do low-THC users interpret the cannabis

items?
• Do they identify as herbal cannabis users?
• May the questions including both be double-
barrelled?

16
Questions with programmed fills

17
Visual clues

This picture shows four different amounts of

resin in crumbled and non-crumbled form.
Which picture shows the amount of resin you
usually put in the [Q17]?

18
Sensitivity issues

Sensitivity of the purchasing questions for

high-risk users

Any drug-production, selling items

20
A tiny bit of theory
In the past 12 months, how often did you use
ecstasy/MDMA?

What is ecstasy/MDMA?
Comprehe
ntion

When was it? Someone’s birthday? And then again on Saturday the
Recall
same weekend…

How much counts as “use”? Does it count as once, or twice?

Judgment

• Not daily but more than once a week?

• Once a week?
Response • Less than once a month?

23
Limitations of cognitive interviews

• Summary of findings may be difficult, or

subjective
• Practices are not standardised, numerous
techniques at use
• No way how to judge how common the
problems are
• Not all problems are subject for cognitive testing

25
How to… fieldwork
Interviewing techniques

 Think aloud
 Verbal probing

 Mixed approach
Think aloud approach

• Respondent asked to describe thoughts

• Externalizing the cognitive processes

28
Think aloud approach

• Requires a respondent training

“Try to visualize the place where you live, and think about
how many windows there are in that place. As you count up
the windows, tell me what you are seeing and thinking
about.”

29
Verbal probing

• To elicit more information

• Probes are respondent-centred (“you”)
• General probes
How did you arrive at that answer? Was is easy, or hard to answer?
• Comprehension/Interpretation
What do you think they meant by ….?
• Paraphrasing
Please, tell me in your words, what the question is asking?
• Recall
How do you remember that?
• Confidence judgement
How sure are you about that?

• Scripted probes vs. spontaneous/emergent

• Concurrent vs. retrospective probes
30
Verbal probing example from the EWSD

How do you usually obtain cannabis resin?

 Buy it from a shop online
 Buy it from a classified ad
 Buy it from internet encrypted markets
 Buy it from a drug dealer
 People give it to me or share it with me for free
 I produce it myself (from home grown cannabis)
 Other (please specify)

Scripted:
What do you think they meant by a classified ad?
And how do you understand the term internet encrypted market?

Spontaneous:
Do you ever buy it from a friend? I mean – do you ever pay to a friend for resin? What option would you
chose here?

31
 Think aloud vs. verbal probing? (Hint: Use both.)

THINK ALOUD VERBAL PROBING

+ less bias introduced by + control

interviewer + no need to train
+ minimal interviewer respondent
training
+ open-ended
- risk of bias & reactivity
- respondent training effect
- resistance & burden - may sound artificial

32
Debriefing session

Overall evaluation
• How did you find the questionnaire overall?
• Was it interesting for you? Did you feel engaged?
• Was it easy for you to answer all questions, or did you
find some of them difficult?
• Too long? Tiring?
• Effort when done online and alone.
• Any other comments?

Other aspects of interest (e.g. slang terms for drugs)

33
Useful tips #1

Experienced interviewers are better.

Interviewers that read well non-verbal clues and don’t take
anything for granted are the best.

Do a mock interview.

Effective communication: shared (well-understood)

expectations!

Some people are natural critics and aloud-thinkers, while

others will always struggle. Accommodate the interview style
as much as possible.

34
EWSD Cognitive Interview Guide

35
Useful tips #2: EWSD questionnaire on paper/screen

Modules
 No need not randomize.
 Aim for one or two drug modules, not more.
 Prepare separate stacks/annotated pdf file to screen.

Routing and skipping patterns

 One question per page where needed

Questions with programmed content

Q23: The last time you used resin, how many
people did you share the [Q17] with?
-> [joint/pipe/water-pipe] 36
How to… before
 Sampling

• Purposive sample of “real” respondents:

• EWSD: different sub-populations of people who use illicit
drugs
• If possible, include also HRDU, lower-EDU subjects
• Sample size: what is feasible / theoretical
• EWSD: at least 10, or 2 per drug module
• Good subject: able to articulate thoughts /
feelings / opinions
• Multilingual where relevant

38
Recruitment

• Informal networks and snowball techniques

 NGOs and other organization providing services to
people who use drugs

• Online: eligibility screening

• Incentives: lunch voucher/online shop voucher

+ motivation and effort
- eligibility criteria

39
Ethical considerations

• IRB may be necessary also for cognitive

testing

• Informed consent – a standard format +

consideration for online interviewing
 Virtual cognitive interviews

• Skype/Zoom – screen sharing; annotation

• Advantages: less time/space constraints; access to distant
respondents
• Disadvantages: tiring; less non-verbal clues; different sense of
privacy; incentives management more complicated
• Additional concerns: audio quality and mode related
problems; disengagement and interruptions
• Recording: face & voice vs. only voice
• GDPR
- Managing personal information
- Consent BEFORE the interview
- Storage limitations
41
 Useful tips #3

• Test the technology & do a mock-interview

• Give instructions to respondents before-hand
• They will need to concentrate for an extended period of time
and respond to sensitive questions
• Private and quiet space
• Use of headphones
• Seek to build a rapport through warm-up
• Consider observer to take notes (also in F2F)

42
How to… analyse and report
Analysis (or rather synthesis?)

Interview summaries
- Useful for contextual information
- Understand why there are problems

Item-by-item summaries
- Problems with particular items

Coding
- EWSD simplified coding

44
EWSD simplified coding

45
Possible outputs

Language & text Structure

- adapting translation - new answer categories

- slang - new items
- formal/informal - re-ordered items/answer
- direct/indirect options

-> decide nationally -> decide jointly

46
EWSD Reporting form

48
Next steps
What’s next?

EMCDDA:
To finalize the questionnaire
Distribute Guide & Reporting Form to countries

Countries:
Check if you need ethical approval
Plan the interviews
Prepare informed consent forms
Translate questionnaire
Seek support if needed
50
How to get in touch

Katerina for IPA7

Katerina.Skarupova@emcdda.europa.eu

Anne for EU4MD

Anne.Bergenstroem@emcdda.europa.eu

João for the EU

Joao.Matias@emcdda.europa.eu

+ working meetings
+ online platform
51
Questions?
RESOURCES

• Willis, GB. 1999. Cognitive Interviewing: A “How To” Guide

https://www.hkr.se/contentassets/9ed7b1b3997e4bf4baa8d4eceed5cd87/gordonwillis.pdf
• Cross-cultural survey guidelines
https://ccsg.isr.umich.edu/
• Willis, GB. 2015. Analysis of the Cognitive Interview in Questionnaire
Design
https://g.co/kgs/xBtwJw
• Beatty, P. C., Collins, D., Kaye, L., Padilla, J. L., Willis, G. B., & Wilmot,
A. (Eds.). (2019). Advances in Questionnaire Design, Development,
Evaluation and Testing. John Wiley & Sons.
https://g.co/kgs/rsZckq
• Presser, S., Couper, M.P., Lessler, J.T., Martin, E., Martin, J., Rothgeb,
J.M. and Singer, E., 2004. Methods for testing and evaluating survey
questionnaire. John Wiley & Sons.
https://g.co/kgs/t1LdWJ