A Pragmatic

Perspective on
Evaluation

Dr. Julie Pétrin

Manager, Impact and Evaluation
MS Canada

March 13th, 2023

 Who am I and how I got to where I am

MS Canada and our Strategic Plan

Layers of evaluation from program to
population level
What’s the Program Evaluation
plan… Operational Evaluation

Organizational Evaluation

Impact Evaluation
Who Am I?
 My Winding Path

BSc Neuroscience

Manager MSc Anatomical Sciences

Impact and Evaluation Post-Doctorate PhD Rehabilitation Sciences
Implementation Sciences
Quick Facts about Multiple Sclerosis
• Inflammatory neurodegenerative condition of the central nervous
system
• Degradation of myelin sheath leading to lesions and axonal transection
• Signs and Symptoms
• Numbness and tingling
• Disease prognosis in uncertain and
• Walking difficulties
difficult to predict
• Fatigue
• Cognition issues • Available treatments and management
• Spasticity
available – however no cure
• Vision problems
• Bowel and bladder dysfunction
 MS Canada
MISSION
A world free of MS

VISON
To connect and empower the MS community to create positive change

CASE FOR URGENCY

Highest 11 Canadians 3.4 Billion
Onset between Women 3X
prevalence diagnosed a annual cost of
20-49 more affected
in the world day MS
MS Canada – Our Teams

Mission Community People

Marketing and Corporate

Philanthropy
Communications services
 People living with MS
People affected by MS
Healthcare Providers
MS Researchers
MS Policymakers and Influencers

Community Volunteers
Donors
Health Industry
Partners
Allied communities and coalitions
Where to start?
Layers of Evaluation within MS Canada
Impact Evaluation

Organizational Evaluation

Operational Evaluation

Program Evaluation
Overview of
Strategic Plan
Impact Framework
 Where are we trying to have an Impact…
Impact Category Description Subcategories
Foster Capacity The MS community needs the people, infrastructure, and People ● Infrastructure ●
philanthropic enterprise to strengthen community and mobilize efforts Philanthropic Enterprise ● Capacity
to achieve its impact goals. for Community to Use MS Knowledge
Advance Knowledge Expand the MS knowledge base with knowledge that can be translated Research Activity ● Community
while providing opportunity for knowledge exchange within the MS Knowledge ● Translatable Knowledge
community and with the public.
Expand Connections Establish, maintain, and strengthen formal (e.g., partnership) and Research & Healthcare Collaborations
informal (e.g., peer support) connections between different ● MS Community Connections ●
community members, locally, nationally, and internationally and Partnerships
expand outside of the MS community.
Provide Support Provide evidence-informed education and resources to people living Education ● Resources ● System
with and affected by MS, the clinical community, and researchers. Navigation ● Support Programs
Inform Decisions & MS knowledge generated and communicated via research, advocacy, Policies ● Practices ● Processes ●
Empower Actions and programs is used to make decisions and empower actions at Behaviours
individual and system levels.
Health Outcomes Changes to/effect on prevention, diagnosis, treatment, palliation of 4 Impact Goals
MS, and health system performance related to the efforts of the MS Advance Treatment & Care ● Enhance
community. Research, programs and services, and advocacy target Well-being ● Understand & Halt
these MS-specific health impacts. Disease Progression ● Prevent MS
4 Impact Goals
• Advance treatment and care: Having a variety of effective treatment
and care options for symptom management, wellness, and self-care
will help people on their unique MS journey.
• Enhance well-being: Removing physical and social barriers within
communities will ensure access to opportunities and supports for
people affected by MS.
• Understand and halt disease progression: Understanding the
complexities of MS progression will stop MS in its tracks.
• Prevent MS: Stopping MS before it starts will reduce the number of
people who develop the disease.
 Single Program Evaluation – Knowledge
Network Logic Model
Goals Provide reliable information, support and referrals in the areas of programs and services,
medications, resources on management, and help in navigating research healthcare services
Inputs • Staffing
• Training
Activities • Training workshops
• Weekly updates on new content/resources
• Call centre, live chat and social media assistance
Audience People living with MS, People affected by MS (friends and family), awaiting diagnosis, Healthcare
providers
Short-term Outcome Improved knowledge about MS and related resources and services
Participants feel supported
Medium- & long-term outcome Informed their decision-making processes
Behaviour change based on knowledge (action)
Process outcomes Reach - number of people reached in each province and across audiences, amount of people
served, Knowledge navigators have access to timely resources, Questions being answered,
provision of accurate info, level of satisfaction
 Single Program Evaluation – KN Data Collection
Objective
Process 2000 individuals reached across
all provinces and territories (per
annum)
Level of satisfaction with KN
Outcome (short) 95% of individuals have
higher levels of MS related
knowledge/awareness
(medium) 75% of people
reporting that they will engage in interaction survey with
an action
(Long) 75% of participants
engage in action
Methods
Quantitative (tracking calls) # of inquiries
Mixed methods (post-
interaction survey with
open ended questions )
Mixed methods (post-
interaction survey with
open ended questions )
Mixed methods (post-
open ended questions )
Mixed methods (post-
interaction survey with
open ended questions )
Indicator Data Source
Tracked in database
% of individuals who were very All inquiries are
satisfied/satisfied provided a survey
link
Comments about their interaction
% of people that Strongly agree/agree All inquiries are
to have gained knowledge/questions provided a survey
answered link
Comments about their interaction
% of respondents that stated they All inquiries are
intend to take action provided a survey
link
Comments about their interaction
% of respondents that engaged in at All inquiries are
least one action (1-month post provided a survey
interaction) link
Comments relating to reasoning
behind action
Example of
Reporting
Operational Evaluation – Mission and
Community Teams

Objective 1. Provide Objective 2. Facilitate

programs that meet connections and
the needs of the MS support for our
community constituents
Objective 1. Provide programs that meet the
needs of the MS community
Key Result 1. Improve user experience and uptake in accessing
our programs
Key Result 2. Establish a renewed program menu informed by
data
Key Result 3. Enhance awareness about MS through launching
a refreshed and reimagines education program that reaches
20% of the MS community
Goal: Provide programs that meet the needs of the
MS community
Objective Activities Audience Process outcomes Short-term Med- long-term
outcomes outcomes and
Improved • Create • Persons with MS # of programs Improved ease of Decrease in number
user improved • Persons affected offered with a hybrid registration of unmet needs
experience registration by MS options (QUANT/QUAL) related to offerings
and uptake in system in PwMS
accessing our • Create # of media views Higher number of
programs hybridized participants in
program programs
offerings
• Marketing Improved overall
towards new satisfaction with
program programs (QUANT/
changes QUAL)
Objective 2. Facilitate connections and support
for our constituents
Key Result 1. Strengthen the network and support of
the MS community by growing constituents'
participation in our activities
Key Result 2. 80% of constituents have an increased
sense of wellbeing by participating in our activities
Key Result 3. 80% of MS Canada led activities will have
a virtual option
Goal: Facilitate connections and support for our
constituents
Objective Activities Audience Process outcomes Short-term Med- long-term
outcomes outcomes and
Strengthen • Create • Persons with MS # of new community • Improved • Increase in
the network community • Persons affected hubs created awareness of number of
and support hubs by MS program offerings volunteers
of the MS • Increase # of toolkits and events
community staffing to developed and • Increase number
by growing knowledge delivered to • Increased # of of touchpoints
constituents' network community hubs participants with MS Canada
participation • Marketing of across programs
in our KN and # of new KN staff and events
activities community
hub # of media views
• Adding
additional 20
social media
posts
Organizational Evaluation
• Objective 1. Accelerate momentum around MS
research
• Objective 2. Deliver value to people affected by MS
• Objective 3. Inspire the MS community to take action
• Objective 4. Ensure the health and growth of the
organization
• Objective 5. Be a data driven organization
Objective 1. Deliver value to people affected by
MS
Key Result 1. Support 20% of people affected by MS through
our programs
Key Result 2. Increase the access of MS Canada information
by 20%
Key Result 3. Improve constituent experiences
Key Result 4. Participate in 2,275 activities to advance our
advocacy priorities
Goal: Deliver value to people affected by MS
Objective Activities Audience Process outcomes Indicator Method
Support 20% At the • Persons with MS Processes are related • # of program • Quantitatively
of people operational • Persons affected to specific activities participants tracked by
affected by level by MS led by teams registrations and
MS through attendance
our programs
Results: Deliver value to people affected by MS – Support 20% of
people affected by MS through our programs
Program Participants N=40,960
45,000

N=40,960
40,000 2022 Year-end Targets
N=36,665
Total 40960
35,000 Target
54000
n=13040
30,000
24%
n=25488
25,000

n=21754
20,000

15,000
n=12,061
n=10473 n=40960
10,000
76%

5,000
n=2279 n=2283 n=2720
n=567
0
Peer support Wellness Education Information and Total
Referral

2021 2022
Impact Evaluation – Impact Assessment
• Impact Evaluation is the sum of the work that an organization
HAS, DOES & PRODUCES

• It is the DIFFERENCE an organization makes at the system level

DOWNSTREAM
Impact is the ‘ripple effect’
that can stem from an
organization’s or community’s
work.
 Where are we trying to have an Impact…
Impact Category Description Subcategories
Foster Capacity The MS community needs the people, infrastructure, and People ● Infrastructure ●
philanthropic enterprise to strengthen community and mobilize efforts Philanthropic Enterprise ● Capacity
to achieve its impact goals. for Community to Use MS Knowledge
Advance Knowledge Expand the MS knowledge base with knowledge that can be translated Research Activity ● Community
while providing opportunity for knowledge exchange within the MS Knowledge ● Translatable Knowledge
community and with the public.
Expand Connections Establish, maintain, and strengthen formal (e.g., partnership) and Research & Healthcare Collaborations
informal (e.g., peer support) connections between different ● MS Community Connections ●
community members, locally, nationally, and internationally and Partnerships
expand outside of the MS community.
Provide Support Provide evidence-informed education and resources to people living Education ● Resources ● System
with and affected by MS, the clinical community, and researchers. Navigation ● Support Programs
Inform Decisions & MS knowledge generated and communicated via research, advocacy, Policies ● Practices ● Processes ●
Empower Actions and programs is used to make decisions and empower actions at Behaviours
individual and system levels.
Health Outcomes Changes to/effect on prevention, diagnosis, treatment, palliation of 4 Impact Goals
MS, and health system performance related to the efforts of the MS Advance Treatment & Care ● Enhance
community. Research, programs and services, and advocacy target Well-being ● Understand & Halt
these MS-specific health impacts. Disease Progression ● Prevent MS
Why Assess Impact?
Accountability
To taxpayers, donors etc.

Analysis

4 ‘s
To better understand what works
and under what circumstances

Allocation
To inform investment decisions
(institution decisions, focus area,
people, etc.)
Jones, M. & Grant, J. (2013). Making the Grade:
Methodologies for assessing and evidencing research
impact. 7 Essays on Impact. Advocacy
To “make the case”
Selecting Indicators
➢ Must include stakeholders in Process
+ Importance FABRIC Criteria
Focused
Appropriate
- Feasibility + Feasibility Balanced
Robust
Integrated
Cost Effective
- Importance
Impact Category: Provide Support
Definition: Evidence-informed MS specific programs are made available and delivered to the MS
community. Providing support influences individuals (PaMS, researchers, advocates, volunteers) and
creates a stronger network to address current and future challenges.

Core Indicator Aspirational Indicator

People served: % of PaMS using resources/programs who feel
• # of people supported → WHEN/HOW
• Regional breakdown
• Type of people Aspects of the programs and services that lead to a
• % growth in participants in programs and services sense of support (Qual)

Quality and consistency of program delivery: Improved levels of self-confidence in managing their
• Overall satisfaction with programs and support condition (When/ How)
• Fidelity measure (?)
• Resolution of issue rate (KN)
• Collaborations with organizations
Impact Category: Inform Decisions and
Empower Actions
Definition: MS knowledge generated and communicated via research, advocacy, and programs is used
to make decisions and empower actions at individual and system levels.

Core Indicator Aspirational Indicator

% of people having used MS Canada information,
services, and programs that have taken an action or
changed behaviors based on a given touchpoint with
MS Canada
# of legislative, regulatory or policies introduced or
adopted at any level of government
# and % growth of individuals from MS community
participating in decision making process
# and % growth of Canadians self-reported
understanding MS risk factors
# of new standardized care practices (e.g. care
pathways, clinical practice guidelines, competencies)
Improved healthcare provider ability to make MS –
related management decisions → Healthcare provider
knowledge, attitude, and behaviour change survey (4
years)
Impact Category: Advance treatment and care
Definition: Improved diagnosis and prognosis, increased effective treatments and care
options, improved holistic disease management, and better delivery and access to
care.
Indicator Metric
Timely diagnosis # weeks from 1st suspected MS-related Healthcare Provider visit to MS diagnosis
Treatment and care options # of Treatments across Canada (available and covered)
Access to Care Wait time to Providers
- Neurologist
- PT/OT
- Mental health providers
- Specialists
# of annual ER and FP visits
Better Care Delivery # of MS clinics that are multidisciplinary
% of PwMS being seen at an MS clinic
Key Challenges of Impact Assessment
• Organizational readiness and philosophy change
• Halo effect
• Attribution
• Double counting
 Developing a 5-year implementation
What’s plan for all indicators across all
impact categories
Next…
Questions???
Thank you