Development and evaluation

Reflections and perspectives – May 2020

In organisations, how do we think about developing capability?
However 70 20 10 is not….
• A universal fix
• A constant ratio
• How all professionals learn
• A set of mutually exclusive categories
• A learning theory
• A criticism of formal learning
• Unfamiliar in the rest of our lives
It is an opportunity to….
• Align to a principle
• Examine what the role of L&D is and extend it
• Connect to business objectives
• Share responsibility for learning
• Build experiential in
• Build social learning in
• Be more flexible about learning
• Recognise that learning happens all the time
Fishbowls…
It is an opportunity to….
Examine what 70 20 10 really says...
• It’s about the individual

• It’s about their peers and relationships, co-workers,

supervisors and managers, direct reports

• It’s about their work, but it’s not extra work

• It’s about the system and environment they are in

APSC Centre for Leadership and Learning evaluation
Framework and examples
Course completions for the period 01/01/2020 to 17/03/2020
Completed Modules
Induction Modules Completions
140
APS Frameworks - Learning Module 131
120
100
Diversity and Inclusion - Learning Module 123
80 Fraud Awareness - Learning Module 114
60 Information Management - Learning Module 107
40 Integrity and Values - Learning Module 110
20 Money and Resources - Learning Module 93
0
Portfolios, Departments and Entities - Learning Module 102
Risk Awareness - Learning Module 93
Security Awareness - Learning Module 92
Structure of Government and the role of Parliament - Learning Module 107
WHS - Learning Module 94
Working in the APS - Learning Module 97
Grand Total 1263

• A total of 1263 modules have been completed since 01/01/2020 for those
No. of Modules Count of Completed new learners who have been uploaded during Quarter 3. The completion
Completed modules
rates across all 12 modules for the induction program are fairly consistent,
0 318
1 18
with APS Frameworks seeing the highest completion rates and Security
2 18
Awareness having the lowest completion rates for the period.
3 10
• Of the 480 learners, 69 have completed all 12 modules with a total of 318
4 5
learners who are yet to complete any of the modules. The low completion
5 4
6 4
rates may be attributed to the fact that 207 new users were only provided
7 5
access to the portal in the last 2 weeks.
8 5
• Of the 69 user who completed all 12 modules, it took an average of 11.9
9 5
10 2
days to complete.
11 17
12 69
Grand Total 480
Total modules completion since commencement of pilot
Our framework is
• Scalable
• Based on investment, novelty, risk, audience
• Comparable across programs and streams
• Agnostic on method
• Surveys, discussions, focus groups, research
• Evidence based
• Other frameworks, experience, practice
• Pragmatic
• Do what we can with what we have
• Focussed on use
• Realistic
• Evaluation data is one piece of evidence for a decision maker
• Accretion
EVALUATION FOCUS Program type Evaluation activity DATA CAPTURED

Impact Selected • End of program action plan – Questionnaire • Action plan: Three actions for
Leadership and capturing participants’ action plan, i.e. individual implementation
[1]
Core Skills goal setting
• Interview: Behaviour change and benefits the
• 6-9 weeks post-program nudge - Email containing program has produced across the APS
participants’ specific actions sent to encourage system (self/team/organisation).
implementation

• 3-9 months post-program participant interview

Implementation Selected • 3-9 months post-program participant interview • Improvement in capabilities attributed to
Leadership and program.
1 • 3-9 months post-program manager interview
Core Skills
• How learning is being implemented in the
workplace, and/or barriers to
implementation.
Capability All • End of program participant questionnaire • Self-report of capability against each
Leadership and program capability measure both before and
Core Skills at the end of the program
Value All • End of program participant questionnaire • Blend of quantitative and qualitative data
Leadership and including: relevance, intention to implement
Core Skills learning, overall value of learning, and
general feedback.
Program design Selected • Various, e.g. observation of program delivery, • Observations
Leadership and interviews with program designers and facilitators,
[2]
focus groups with participants, examination of • Interview transcripts and data
Core Skills
program artefacts. • Findings and recommendations for
improvement
Our framework is
What we have learned
• Political will
• Resources and actions
• Principled
• Use frameworks, methods, tools that are informed by your values
• Programmatic
• Other frameworks, experience, practice
• Pragmatic
• Aim high, but do what you can
• Persistence
• Takes time to embed, be patient
• Practice
• We are refining as we go, learning from others and our experience