Troubleshooting in Selected

Qualitative Methods for Evaluation

[insert date]
Qualitative Methods
in Evaluation of Public
Health Programs
Session 3
The evaluation process
Session objectives
At the end of this session, participants
will be able to:
 Explain the pros and cons of qualitative methods
for rigorous evaluation

 Describe methods to mitigate common problems in

qualitative evaluation
Qualitative inquiry
Selected methods
Qualitative inquiry
Selected methods

Observations

In-depth interviews

Focus group discussion

 Observations
Opportunities
 See program operations and
interactions with program
officers and beneficiaries first
hand
 Record information
in addition to what participants
tell you
 Describe settings, interactions,
and activities in detail
 See everyday details
that you might not otherwise
capture
Challenges
 Requires time and patience
 May need consent for doing
observations
 Participants may change behavior
in your presence
 Selective perspective of
the person observing
 Characteristics of the observer—
gender, culture, etc.
 Standardization of accounts from
different evaluators
Observations
Issues of standardization
 Use observation guides or structured/
semi-structured protocol
 Teams may choose to approach observations
with target concepts
and criteria to observe
 By working in teams, field staff can review one
another’s field notes and help identify objective
versus subjective observations
Group Activity 1
Observations
 Observe role play
 Take detailed notes on what you observe
o What do you see? Hear? (no interpretation)
o Interpretation

 Determine the role of the observer

(active participant or outside observer)
 Plenary discussion (10 minutes)
Scenario
A community health worker (CHW) is visiting Kavita,
who is pregnant with her second child. Kavita is in her
second trimester but has not been had antenatal care
(ANC). Kavita is undecided on whether she should go to
clinic for ANC. She did not get ANC with her first child
and he’s doing fine. She is very busy at home cooking,
cleaning, and taking care of her two-year-old son. Her
husband, Krishna, does not want her to go. He prefers that
she stay at home taking care of her household duties.
 In-depth interviews
Opportunities
 Open-ended discovery and
understanding between
interviewer
and participant
 Yield rich and detailed
information
 Opportunity to establish
rapport with participants and
ask follow-up questions
Challenges
 Time intensive
 Requires interviewer skills to
elicit high-quality information
 Challenges of individual
participants’ characteristics and
building rapport
 Participant and interviewer
reactions
to sensitive topics
In-depth interviews
Tips and techniques
Basic techniques
 Don’t ask leading questions
o “Do you think people in the school community don’t talk
about sex and condoms because they might be seen as
promiscuous?”
 Avoid judgement
 Tolerate silence
 Avoid giving nonverbal cues
Useful phrases
 “Tell me more.”
 “How did that come about?”
 “Uh-huh…” [or nodding head]
In-depth interviews
Useful techniques
Probe for details:
“When did that happen?”
“How did it come about?”
“Who was involved?”

Use clarification probes:

“You said the program has been a success.
“Please say more about what you mean
by ‘success.’”
 In-depth interviews
Participant The challenge Techniques to address
characteristics or mitigate

Very talkative Continues to talk; but  Redirect, use specific

not about the scenarios or stories
interview topic  Maintain control and ask
or provides too much specific questions
detail to enhance responses

Very shy and/ Is not forthcoming  Use supportive or

or reserved with information; recognition responses
is not responsive to  Be mindful of
questions the dynamics
Group Activity 2
With partners
With a partner, decide who will be the participant and who
will interview (5 minutes).
 Participant role: Choose to be “very talkative”
or “very reserved”
 Interviewer: Practice techniques; probing
 Topic should be personal, but your choice; examples:
o What influenced your decision to come
to this training?
o What led you to working in public health?
o How did you decide what clothing to wear today?
 Flip roles (5 minutes)
 Plenary: Discuss as a group (5 minutes)
Group Activity 2
Plenary discussion
 How did the results of probing compare with the
initial answer to the question?
 How did it feel as a respondent
to be probed?
o Did you feel that probes helped you
to give better information or not?
 Did interviewees sense how the interviewer felt about
the answers? Or, whether they could tell what the
interviewer was looking for with certain questions?
 Focus group discussions
Opportunities
 Cost-effective data collection
 Identify major themes
 Hear from multiple participants in
one session
 Participants define what
is important
 Some opportunity to
explore issues in-depth
 Possibility to follow-up
and clarify
Challenges
 Group dynamics must
be carefully considered (including
gender
and minorities)
 Limited ability to collect detailed
and in-depth information
 Limited control over the process
or conversation
 Not suited to collection
of sensitive information
 Limited control over
confidentiality
Focus group discussions
Useful techniques
 Limit the number of questions you plan to ask (this will
vary by group size and time)
 Pilot your questions and topic guide
 Maintain focus on the topic
 Manage the process so that all participants are able to
share views
 Carefully consider group dynamic
o May need to separate by sex, age, etc.
 In general, avoid highly personal or controversial topics
 Recognize cultural or contextual barriers
 Focus group discussions
The scenario The challenge Techniques to address
or mitigate

One participant One, or a few participants, are  Acknowledge their contribution

dominates the quick with answers; they may and ask
conversation. dominate conversation or be for responses from
argumentative other participants
with others.  Smaller groups
You notice Participants may lack confidence  Use supportive or recognition
specific to talk in front responses
participants of each other—may feel threatened  Be mindful of
aren’t talking. by the presence of others or the the dynamics
moderator.
Participants Lack of discussion  Consider the dynamics
simply agree with or rich details. of the group
one another.  Use probes to further engage
participants
Group Activity 3
Focus group discussion
 In groups of 5–7, you will conduct a mock focus group
with the materials provided
 Each participant will choose a “role”
from the envelope
 Facilitators will use the topic guide
(20 minutes)
 Switch roles halfway through
 Plenary discussion (5 minutes)
Participant roles (Activity 3 continued)
1. The dominator: Tries to assert authority or superiority in manipulating the
group or certain members of the group (interrupting the contributions of
others, etc.)
2. The opinion-giver: Wants to state her/his beliefs pertinent to
a suggestion made (the emphasis is on what she/he believes should be the
group’s view of pertinent values)
3. The aggressor: May deflate the status of others, expresses disapproval of
the values, acts, or feelings of others
4. The blocker: Tends to be negativistic and stubbornly resists, disagreeing
and opposing without or beyond "reason"
5. The distractor: Changes direction of the conversation
6. Shy Sal (or shy Sally): Does not say much, looks very shy,
does not keep eye contact, speaks in a low voice
7. The self-confessor: Uses the audience opportunity which the group setting
provides to express personal, non-group oriented "feelings," "insight,"
"ideology," etc.
Summary
Take home messages
 Be aware of yourself as the observer, interviewer, or
moderator
 Consider context, culture, gender,
and power dynamics for any setting
 Develop test guides or other tools prior to use (next
session gets into tool development!)
 Choose the method best suited to your evaluation aims
This presentation was produced with the support of the United States
Agency for International Development (USAID) under the terms of
MEASURE Evaluation cooperative agreement
AID-OAA-L-14-00004. MEASURE Evaluation is implemented by the
Carolina Population Center, University of North Carolina at Chapel Hill in
partnership with ICF International; John Snow, Inc.; Management Sciences
for Health; Palladium; and Tulane University. Views expressed are not
necessarily those of USAID or the United States government.

www.measureevaluation.org