You are on page 1of 19

Safety KPI – what for?

Learnings from the attempt to build a
meaningful Safety Management Cockpit
René Aebersold, Deputy Chief Safety Officer skyguide

Table of Contents 2 1 Why measure safety ? 2 What to measure? 3 How to measure ? 4 What to do with the measured data? 5 Conclusions .

› To identify trends. › To compare with others. too… › To punish. Hooray! › Because others do it.Why measure safety? › To prove how good we are…. › To improve. › For the money (to get a bonus). 3 1 .

Why measure safety? Goal of a Safety Management System SMS: Continuous safety improvement Corrective actions from analysed safety performance monitoring data OIR SIR Audit findings 4 Management Lessons learned Measures for Improvement Staff .

What to measure? 2 Reactive indicators: › To measure the (long-term) impact of our safety efforts › To identify hotspots and trends › To verify/validate assumptions made in pre-assessments Difficulties with reactive indicators in ATM: › Very low numbers for significant events (low probability) › Direct allocation of impact to efforts › Creates "bad" feelings and urgency to immediately react › In most cases only negative data is measured 5 .

possible problems › To identify drifts into failure Difficulties with proactive indicators in ATM: › Difficulty to argue direct relevance.What to measure? Proactive indicators: › To measure safety efforts (output) › To detect week points. to justify measures › Information might be biased › Building relevant proactive indicators is difficult 6 2 .

Investigation and Improvement SA9 Safety Performance Monitoring SA10 Ops.example for proactive SPI 2 SPI 1a . Safety Survey and SMS Audit Safety Promotion SA11 Adaptation and sharing of best practices .What to measure .Safety Maturity Index according ECTL SRU SA1 90 SA11 SA2 80 SA10 SA3 70 60 SA4 SA9 SA8 SA5 SA7 2011 7 SA6 Interquartile range Agenda: Safety culture SA1 Development of a positive and proactive safety culture Safety Policy SA2 Org. and individual safety responsibilities SA3 Timely compliance with international obligations Safety Achievement SA4 Safety standard and procedures SA5 Competency SA6 Risk Management SA7 Safety interfaces Safety Assurance SA8 Safety Reporting.

What to measure? Sharp-end indicators: › Measuring failures directly at the impact point Difficulties with sharp-end indicators in ATM: › Have a high potential for blaming operators › Blaming or even punishing the operators will reduce the number of reports and the effectiveness of the SMS 8 2 .

Workplace Management Company Authority Government Blunt end: unwanted outcomes happend earlier and somewhere else Morals. blunt-end Activity Hollnagel. social norms What to measure? Sharp-end vs. 2004 9 2 Sharp end: unwanted outcomes happen here and now .

the more difficult it gets to take corrective actions (no direct control) 10 .What to measure? 2 Blunt-end indicators: › Measuring things which can contribute to sharp-end failures Difficulties with blunt-end indicators in ATM: › Sometimes difficult to measure › Sometimes difficult to allocate the direct relevance › The further away we go from the sharp end.

91% of all SSG and SSGSubgroup members or representatives were present during the 2nd quarter 2012. planned 80% 3c Attendance SSG and SSG-Subgroup (presence members or deputy) 80% 3d Training sessions in SSG and SSGSubgroup meetings as planned in yearly program 80% 3a 11 2 Q3 Level reached Q4 Q1 Q2 100% 100% 100% 100% 100% 100% 100% 100% 78% 72% 100% 80% 91% 100% 50% Review 50% Target 2012 nil 100% nil 80% In average.realised vs. 80% .example for blunt-end SPI SPI 3 SPI Management attention for safety Definition Target 2011 SSG and SSG-Subgroup meetings realised vs.What to measure . planned 100% 3b EM meetings: 5 times 30 minutes and one time 60 minutes dedicated to safety matters . 80% 1 of 2 trainings have been hold during the 2nd quarter 2012.

What to measure? 2 "Hard" or direct indicators: › Probabilistic or quantifiable indicators Difficulties with "hard" or direct indicators in ATM: › Do we count the right things? › Relatively easy to apply in technique. human factors 12 .g. much more difficult to apply for e.

What to measure? "Soft" or indirect indicators: › To measure symptoms › To get a feeling for the consequences of actions › To "measure" safety culture Difficulties with "soft" or indirect indicators in ATM: › Considerable effort needed (lots of people involved) › Individual measurement may not be relevant. but trend analysis are -> long term indicator › Needs an open and trustful culture 13 2 .

example for "soft" SPI 5 Target Group Survey Results 2Q 2012 (SPI 2a) 4 3 2 1 Mittelwert Skalen 2 .14 et e rs / r e so ur c rs M rn ur c te es al Cu lt u re so in St ra te SM Ri sk s ot iv a t io n Pr oc es se s an re et e es ex t gy A b er ilit na y t l o re A b sp ilit on y t d o A b m ilit on y t it o o r an t ic Pr ip a A b eo te ilit cc up y t at o io le a n w rn Re it h lu c fa ta ilu nc Se re e t ns o s it i v it im y t Co pl o if y m op m it m er at en io tt ns D o ef M r er es in en df ilie ul c nc e en e to es ex s o p rg er an t is iz i e ng sc al e Pa ra m H u m Pa ra m S What to measure .

15 S-MIS Resilience engineering HRO Mindfuleness organizing scale Deference to expertise Commitment to resilience Sensitivity to operations Reluctance to simplify Preoccupation with failure Ability to learn Ability to anticipate Ability to monitor Ability to respond Parameters / resources external Strategy SMS Risks Processes Motivation Culture Human resources Parameters internal What to measure .example for "soft" SPI 2 Target Group Survey (N25 deviation against basis N104) (SPI 2a) 20% Change of Model Parameter after 1st run (2Q 2012) 15% 10% 5% 0% -5% -10% -15% -20% .

occurrence reports 16 3 .How to measure? › Safety Performance Indicators (KPIs): › are measuring the output of our safety efforts › mostly proactive. safety improvement reports from staff. sharp end › tools are "been counting" data collections. analysis Safety indicators (SI): › are measuring the safety impact of our safety efforts on our system › mostly reactive. blunt end › tools are surveys. indirect. direct.

17 . › One measurement cycle is very rarely relevant – but trend analysis are.What to do with the measured data? 4 › We measure a lot of data – which must now be turned into information. or react too late. the more difficult it becomes to argue for taking corrective actions. through proper analysis. over longer periods. therefore it is important to stabilise the KPIs. › The further you measure towards the blunt end. involving operational experts. › The biggest difficulty is to decide when to take corrective actions – you can over-react. › Very often you will not find very significant variations – but this is also a relevant information.

We have to measure different sets of safety indicators: reactive – proactive. Expectations are high – results sometimes not very significant. We have to use a mix of measurement techniques. but still relevant. The effort is considerable! The most difficult part is the transition from measured data to information which is useful for improvement. 18 . sharp-end – blunt-end. direct – indirect.Conclusions: Why do we need Safety KPIs? 5 We need to measure safety indicators in order to be able to improve more effectively.

Questions? 19 .