Service Level Management With Apdex

Peter Sevcik Executive Director, Apdex Alliance President, NetForecast peter@netforecast.com 434 249 1310

Outline

    

Apdex Overview Apdex Parameters Apdex Scope Apdex in Service Level Management Apdex SLM Case Studies

© 1990-2010 Peter Sevcik and NetForecast, Inc., All rights reserved.

©2010, Apdex Alliance Inc., all rights reserved.

Slide 2

Calculate the Apdex formula 6.91 [6] Score T Unacceptable 0. and  Enterprises ©2010. all rights reserved.94T 0.00T 0. Apdex Alliance Inc. Apdex Alliance Inc. Define T for the application T = Target time (satisfied-tolerating threshold) F = 4T (tolerating-frustrated threshold) 2.  Measurement approaches. Display Apdex score showing T 7.Apdex Defined  Apdex is a numerical measure of user satisfaction with the performance of enterprise applications  It defines a method that converts many measurements into one number  Uniform 0-1 scale  0 = no users satisfied  1 = all users satisfied  Standardized method that is a comparable metric across  Applications.. Optionally display score using quality colors 2 Existing Task Response Time Measurement easu e e t Samples 7 Excellent Good Fair 1.. Count number of samples in three zones 5.85T Report Group: Application User Group Time Period 3 0. Extract data set from existing measurements 4. all rights reserved. Define a Report Group 3.00T . Slide 3 How Apdex Works 1. Slide 4 0.50T F Tolerating 1 T Satisfied ©2010.70T Frustrated 5 4 Tolerating Satisfied + 2 ApdexT= Total samples Poor 6 0.

Apdex Alliance Inc. ©2010. All rights reserved. Outline      Apdex Overview Apdex Parameters Apdex Scope Apdex in Service Level Management Apdex SLM Case Studies © 1990-2010 Peter Sevcik and NetForecast. all rights reserved. Apdex Alliance Inc. so it looked like all was well But: Apdex = 0...Case Study: B-to-B Commerce Website 16% Major eCommerce site ($4B annual on-line sales) North American broadband users accessing the San Francisco data center Probability of Experiencing the Time y 14% 12% 10% 8% 6% 4% This site had an average response time of 4 seconds. Inc. all rights reserved. Slide 6 .7310 = Fair 52% Satisfied 42% Tolerating 2% 6% Frustrated 0% 0 20 40 60 80 100 120 Slide 5 Load Time of a Typical Business Page (sec) ©2010..

Process for Setting the Apdex T Understand Application Tasks This Webinar Covers Determine Some T Markers Ongoing Measurements (at least 1 month) Calculate Apdex Reports Select Initial T T Observe Users. Business. all rights reserved. Apdex Alliance Inc. all rights reserved. Slide 7 What is a T Marker? Performance Zone Boundaries 0 T Tolerating M2 M3 F (F = 4T) Satisfied M1 Frustrated    A T Marker (TM) is an indicator of the user satisfied-tolerating boundary for an application Some T Markers have a greater than or less than property Multiple T Markers can be used to converge on an initial T value Slide 8 ©2010. .. Events Assess Good Vs. Bad Periods Adjust T No Do Apdex and observations correlate? Yes Validated T & Early Apdex Score Results ©2010.. Apdex Alliance Inc.

3 2.6 5 2 5 7 9 ©2010.9 6. Controlled Performance Experiment 7.4 8 2 6 16 9 6.4 1.7 10. Frustration Indicator 9.9 9.7 10.3 12. Mathematical Inflection Point ©2010.8 6.3 2. Apdex Alliance Inc.7 5. Best Time Multiple 8. Empirical Data 3. Slide 9 Major Website Case Study: Initial T Selections T Markers From Three Sources Circle # indicates setting T method Initial T Selection 9 Survey Targets 10 Measured (T Inflection) 3 Behavior Model Voting T Markers Average Consensus Initial Ts Home Search My Page Shopping Community 6..0 4.T Marker Alternatives 1.4 4. 3 User Behavior Model 4.. all rights reserved. Apdex Alliance Inc.5 2. Source: NetForecast Slide 10 . all rights reserved. Interview Stakeholders 10. Outside References 5. Observing the User 6. Default Value 2.

Slide 11 Where Apdex is Applied Applicable measurements exist in each green cell Desktop WAN (private or Internet) Data Center Edge Web Server App Server Database Server Task Turn Packet X X X X X X X X X X X (DB call) Apdex is already being applied in each cell with an X ©2010. all rights reserved. Apdex Alliance Inc. Inc...Outline      Apdex Overview Apdex Parameters Apdex Scope Apdex in Service Level Management Apdex SLM Case Studies © 1990-2010 Peter Sevcik and NetForecast. All rights reserved. Slide 12 . all rights reserved.. ©2010. Apdex Alliance Inc.

Slide 14 .8 0.1 – 0. Slide 13 Extending Apdex Beyond Response Time  Fundamentally Apdex is a way to present performance  Represented by many measurements  Of a service function  That can be grouped into 3 performance zones  The formula can be applied to many services       Response Time: Packet. Apdex Alliance Inc.1 – 0. Apdex Alliance Inc. or Task Bandwidth Loss Jitter Other infrastructure metrics Business metrics ©2010..1 0.2 – 1. all rights reserved.0 0 Values supplied by Apdex members ©2010. Turn.8 0. all rights reserved.8 0.0 0.01 0.05 0 01 .3 01–03 0..Different T Values at Each Tier Typical Apdex T in seconds Desktop WAN (private or Internet) Data Center Edge Web Server App Server Database Server Task Turn Packet 2 – 12 2 – 10 0.1 – 0.

Apdex Alliance Inc.. .95 [8] 25% 20% 15% 10% Frustr.. all rights reserved. Tolerating Satisfied F F = T/4 T 5% 0% ‐ 2  4  6  8  Bandwidth (Mbps) 10  12  Slide 16 ©2010. Apdex Alliance Inc. all rights reserved.Apdex Applied to Bandwidth 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% ‐ 2  4  6  8  Bandwidth (Mbps) 10  12  Slide 15 ©2010. Apdex Applied to Bandwidth 45% 40% 35% 30% Apdex = 0.

All rights reserved. Inc. Apdex Alliance Inc. Source: NetForecast Slide 18 ... all rights reserved. ©2010. Slide 17 Performance Management Layers Link Communicate Measure Understand Best Practices Assurance Capacity Availability Incident Processes Root Cause Agents Discovery MIBS Diagnostics Logs Alarms Measurements Many More Analytics Data Storage Instrumentation Tools ©2010.Outline      Apdex Overview Apdex Parameters Apdex Scope Apdex in Service Level Management Apdex SLM Case Studies © 1990-2010 Peter Sevcik and NetForecast.. all rights reserved. Apdex Alliance Inc.

.Service Level Management as Defined by ITIL  SLR – Service Level Requirement Service Assurance is a Ra ange  What the business needs  SLO – Service Level Objective  Business need in IT terms  SLT – Service Level Target  Goal in measurements of the IT terms Different types of reports are applied along this spectrum  SLA – Service Level Agreement  Delivery terms and conditions added ©2010. Slide 20 . all rights reserved. Slide 19 Apdex Mapped Into ITIL SLM  SLR – Service Level Requirement  Apdex methodology defines 10+ ways to map requirements  SLO – Service Level Objective  This is Apdex T  SLT – Service Level Target  This is the Apdex score that you should be above  SLA – Service Level Agreement  This is the window of time when the Apdex score matters Apdex T is not the SLA! ©2010. Apdex Alliance Inc. Apdex Alliance Inc.. all rights reserved.

. ©2010. Slide 22 . All rights reserved. Slide 21 Outline      Apdex Overview Apdex Parameters Apdex Scope Apdex in Service Level Management Apdex SLM Case Studies © 1990-2010 Peter Sevcik and NetForecast.. Apdex Alliance Inc. Apdex Alliance Inc. Inc.. all rights reserved. all rights reserved.Apdex in Each Part of ITIL SLM Your organization will need to implement a process to work down this table This Webinar Covered KPI SLR SLO SLT S SLA From 10-ways to define T Apdex T Apdex Score Goal Percent Sco e Goal is met e ce t Score Goa s et Report Initial T agreement Formal T agreement Report of Apdex scores Apdex-based SLA report pde based S epo t Apdex is listed in the KPI Library See http://kpilibrary.com/ ©2010.

75 0.70 0 70 0..80 0.90 0.60 SLA Weekdays 0.Case Study: Web Service Apdex Report California 1.. 3/1 3/2 3/3 3/4 3/5 3/6 3/7 3/8 3/9 3/10 3/11 3/12 3/13 3/14 3/15 3/16 3/17 Slide 24 Source: NetForecast .55 2/25 2/26 2/27 2/28 3/1 3/2 3/3 3/4 3/5 3/6 3/7 3/8 3/9 3/10 3/11 3/12 3/13 3/14 3/15 3/16 3/17 Slide 23 ©2010. all rights reserved.85 Apdex [4] SLO 0. all rights reserved. Source: NetForecast Not Using Apdex: Average With Threshold California 24 Colorado Florida Minnesota New York 20 Average (sec) SLA 16 Weekdays 12 SLO & SLT 8 4 0 2/25 2/26 2/27 2/28 ©2010. Apdex Alliance Inc.95 0 95 Colorado Florida Minnesota New York SLT 0. Apdex Alliance Inc.65 0.00 0.

Apdex Alliance Inc. 5 days per week.. Apdex Alliance Inc. all rights reserved. Source: NetForecast Slide 26 . 3 weeks in the report = 75 state-days) Apdex SLO SLT SLA SLA Compliance = 4 sec >= 0. Source: NetForecast Slide 25 Very Different SLA Compliance Reports SLA compliance report is the percentage of days that the SLA was met (5 states. all rights reserved.9 Score Weekdays 76% Averages <= 4 sec N/A Weekdays 88% Deviation from Normal >= 50% N/A Weekdays 93% ©2010..Not Using Apdex: Deviation From Normal California 20 Colorado Florida Minnesota New York Deviation of 50% From Normal (sec o c) 15 SLA 10 SLO & SLT Weekdays 5 0 (5) 2/25 2/26 2/27 2/28 3/1 3/2 3/3 3/4 3/5 3/6 3/7 3/8 3/9 3/10 3/11 3/12 3/13 3/14 3/15 3/16 3/17 ©2010.

all rights reserved.945  SLT 0.. Apdex Alliance Inc.920  ©2010.925  Good Performance 0. Apdex Alliance Inc.950  0. Slide 27 e-Government Portal Apdex Daily Report 0.955  SLO Apdex [5] 0. Slide 28 .935  SLA 0. all rights reserved..940  0.960  Excellent Performance 0.930  0.Case Study: e-Government Portal ©2010.

. Apdex Alliance Inc. Sep 19 11‐Sep 13‐Sep 15‐Sep 17‐Sep 19‐Sep 21‐Sep 23‐Sep 25‐Sep 27‐Sep 29‐Sep Slide 29 0% 1‐Sep 3‐Sep 5‐Sep 7‐Sep 9‐Sep ©2010.e-Government Portal SLA Compliance Report (Hours When Apdex score >=0. Sep 18. Source: NetForecast Slide 30 . and Friday.94 in 8AM to 8PM Period) x 8 100% Daily SLA Compliance 90% 80% 70% 60% Percent of Time SLA is Met 50% Trend Line 40% 30% 20% 10% SLA Summary: The portal was not 100% compliant for any day in September The average compliance was 70% The trend is negative Serious performance degradation occurred on: Tuesday. Apdex Helps Performance Management Selection of T and the Apdex score is a link to business needs Facilitates technologist and non-technologist communication Defines measurement taxonomy Encourages collaborative dialog Link Communicate Measure Understand This Webinar Covered Foundation for meaningful SLAs Trend analysis finds capacity problems User-driven availability Discovers hidden problems Best Practices Assurance Capacity Availability Incident Processes Root Cause Agents Discovery MIBS Diagnostics Logs Alarms Measurements Many More Analytics Data Storage Instrumentation Tools ©2010. all rights reserved.Thursday.. Sep 16. Apdex Alliance Inc. all rights reserved.

apdex. first articles published in Jul 2002 and Nov 2003 Alliance founded: 2004 Alliance structure: non-profit organization Objective: Define and promote effective performance reporting methods Technology: Open standard freely available to all Sponsors: 27 companies since 2004 Members: 2.000 individuals interested in following and implementing Apdex .Questions and Answers Go to www.org/blog/ Apdex Alliance Facts Invented: by Peter Sevcik.org for more information Join the dialog at the Apdex Exchange http://apdex.