Where’s the Evidence?

Measurement Challenges

Angela A. Aidala

Columbia University Mailman School of Public Health

Columbia Center for Homelessness Prevention Studies

HOUSING AND HIV/AIDS RESEARCH SUMMIT V

June 2, 2010 – Toronto,Ontario
 Presentation Goals
Discuss data collection methods and tools for:
 Describing/ documenting programs
What is the program about? What does it do?
 Describing/ documenting contexts
What are risk promoting or protective contexts?
 Answering process questions
How did it happen? How does it operate?
 Answering outcomes questions
So what? What difference did it make in the lives of
individuals, families, communities?
 Credible Evidence

“Are you just pissing and moaning, or can you verify what you’re saying with data?”
 Collecting Data
Different data collection strategies are
appropriate for answering research questions
 Surveys / interviews - structured conversation using
a questionnaire or interview guide
 Focus groups (group interviews)
 Records review (staff reports)
 Administrative data (surveillance indicators)
 Biomedical assessment (x-rays, blood tests)
 Ethnography, observational analysis (being there)
 Environmental assessments (material, social context)
 Measures
 Most of what we want to study cannot be directly seen,
heard, touched, etc.
- Racism, stable housing, good health
 Need procedures to define concepts so that they can be
measured
- ‘Stable housing’ = same address 6+ months
 Need to be RELIABLE (consistent result when use tool)
 Need to be VALID ( is true measure of phenomena)
 Need to be appropriate to answer research question
 Need to be appropriate for different populations
 What Should be Measured?
 Contexts: Social, Economic, Political, Cultural, Service

Census data
City/county data
HIV surveillance
Service directories
Newspaper archives
Informant interviews

 Need to consider: People’s perceptions and experiences

 Useful tools: HUDuser.gov, American FactFinder

 Ex: Distressed Neighborhood
Neighborhood operationally defined as block
group – “distressed” indicated by summary %
 households below 75% of the poverty line
 vacant housing units
 female-headed households with children < 6yrs
 unemployment rate
 public assistance income
 crimes against persons per 1,000 population
 property crime per 1,000 population

Lee (2009). Factors Associated With Departure From Supported Independent Living Programs
 What Should be Measured?
 Process data: Information about how a program
works and lessons learned implementing the program
that could guide replication or expansion
Routine agency documents
Informant interviews
Meeting notes
Observations
 Useful strategy: Month-by-month chronology
 Useful tool: Standard form for meeting notes
Ex: Record of Issues and Decisions
 What Should be Measured?
 Outcome Indicators: Specific items of information
that track a program’s success on outcomes
 Levels of outcomes can focus on any of:
-- Individual (client)
-- Agency, organization, program
-- Group, social network (family, local club scene)
-- Service System
-- Broader Community

10
 What Should be Measured?
Outcome indicators can focus on:

Desired Change: Types of Outcomes:

Attitudes Improved self-efficacy
Perceptions Increased perception of need
Knowledge Greater awareness of services
Skills Improved problem solving
Behaviors Increased condom use
Conditions Reduction of housing instability

11
Interviewing Clients for
Service Delivery AND Research

Does Not Mean: Does Mean:

 Staff cannot have their  Intent of questions must be
own style understood
 Information restricted to  Systematic way to gather
check lists information
 All information must be  Information about the same
obtained at one time client must be linked
 Staff cannot establish  Interviewers cannot lead
rapport with clients clients to answer

12
 Some Rules
 There should be good reason for asking each
question
 Use established questions and scales where
appropriate – e.g. SF-12, CDQ
 Begin with topics the client might want to discuss
NOT demographics
 Provide interviewer instructions on form
including ‘skip’ directions
 For closed questions make sure all alternatives
are presented
 No ‘double barrel’ questions
 Shortest is not always best !
14
 HOPWA INTAKE FORM
Date:_______________________ Intake Worker:____________________________________
Referral
Source:_______________________________________________________________
Legal Name:________________________________________ Other
Name_______________
Phone:_________________________________Beeper:_______________________________
Current Address:______________________________________________________________
___________________________________________________________________________
_
Last Known Address___________________________________________________________
___________________________________________________________________________
_
Date of Birth:_________________ Age:_______ Place of Birth: City____________________
State:________________ Country:__________________ U. S. Citizen: Y N
Sex:________________ Sexual Orientation:_______________ SS#____________________
Race:_____________________ Ethnicity:___________________
Religion:_______________
Emergency Contact:
Name/Relationship: _____________________________________ Phone:________________

PRESENTING PROBLEM AND SERVICES REQUEST:

___________________________________________________________________________
_
___________________________________________________________________________
_
___________________________________________________________________________
_

Other agency involvement:______________________________________________________

 Client ID # |___|___|___|___|

HARBOR OF HOPE PROGRAM ASSESSMENT FORM

Today’s Date:____ /____/____ Interviewer:________________________

mm dd yyyy
1. What is your name ________________________________________________

1a. Preferred name ________________________________________________

2. What brings you here today? ( State in client’s own words):

____________________________________________________________________
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________
_____
Services Requested (circle all that apply)
01...Shower 06...Birth control 11…Job Training
02...Food 07…Condoms 12…Public Assistance
03...Clothes 08... Counseling, MH 13...Medical services
04...Emerg’y Shelter 09…AOD services 14...Medicaid
05...Permt Housing 10...GED, education 15…HIV testing
 We’ll talk more about these issues later in the questionnaire but first some
TYPICAL QUESTION ABOUT CLIENT HEALTH:

Health:______________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________

STANDARDIZED HEALTH MEASURE:

Next are some questions about your health at the present time.
1. In general would you say your health is. . .
Excellent. . . . . . . . . . . 1
Very Good. . . . . . . . . . 2
Good. . . . . . . . . . . . . . 3
Fair . . . . . . . . . . . . . . . 4
Poor. . . . . . . . . . . . . . . 5

2. The following items are about activities you might do during a typical day. Does your health
now limit you in these activities . . .

17
Indicators: Physical Health

 CD4 Count provide categories (i.e. below

200, above 500)
 Viral load –undetectable v. detectable
 Score on health functioning measure:
MOS –SF12
 General health question: Health now
excellent, very good, good, fair, or poor.

18
Indicators: Mental Health

 Client under-recognition of need for mental

health services
 Systematic screening v. ‘seems depressed’
 Use standardized measure as well as self-
report of “problems”
 Diagnostic screeners (e.g CDQ) provide more
information than symptom checklists

19
Engagement with Medical Care

 “Usual source of medical care” can be

ambiguous
 Need also ask: How many visits for HIV care
from a doctor or other medical provider did
you have in the past 6 months?
 ER visit, hospitalized over night or longer in
the past 6 months
 Stopped going to the doctor for any reason
for 6 months or longer

20
ASKING ABOUT SENSITIVE TOPICS

 Provide brief introduction to question and give

assurance of confidentiality

 Ask whether client has ever engaged in

behavior before asking about current behavior

 Ask “how often” or “when was the first” rather

than “did you ever”

 Staff discomfort with questions will make client

uncomfortable

21
E. SUBSTANCE USE

 Many people have experiences with alcohol and use of different substances. We would like
to know about your experience as part of your health profile. I’d like to remind you that
everything you say is strictly confidential and protected

5. Have you ever used any of the following, even one time...
Go down entire list, then go back and for any YES ask: Did you use (drug) in the past six months?

Used past six

Ever used months
SUBSTANCE Yes No Yes No
1. Marijuana, hashish 1 0 1 0
2. Powdered Cocaine 1 0 1 0
3. Crack (rock) 1 0 1 0
4. Heroin or speedball 1 0 1 0
5. Amphetemines or methanphetemines 1 0 1 0
6. Inhalants , poppers 1 0 1 0
7. Any other drug used regularly, 1 0 1 0
3+ times wk or more often:
Drug: _____________________

22
DATA QUALITY ASSURANCE

 All persons using forms must be trained

 Don’t assume knowledge or comfort level of
staff
 Use training documents e.g. QxQ
 Pilot test data collection with consumers
 Review completed forms for clarity and
completeness

23
My question is, “Are we making a difference?
 RESOURCES FOR MEASURES
 Tools for HOPWA Outcome Based Evaluation
http://www.aidshousing.org

 Community Toolbox - Resources for Healthy

Communities http://ctb.ku.edu/en/Default.htm

 Columbia Center for Homelessness Prevention

Studies - Measures Library http://www.columbia-
chps.org/resources/Homelessness-and-Housing-
Measures

 MOS-SF12 v2 http://www.sf-36.org/tools/sf12.shtml

 CDQ & Training Manual aaa1@columbia.edu

 HRSA, CDC, SAMHSA online guides

25