Metrics & Dashboards

Survey Results
With help from Marty Klubeck at Notre Dame and Brenda Osuna at USC

Who are we?

Brown Carnegie Mellon Columbia Cornell CU-Boulder Duke University Georgetown University Harvard* Michigan State University New York University Penn State Princeton Stanford University UC San Diego UCSF University of Chicago* University of Iowa University of Michigan University of Minnesota University of Notre Dame University of Washington University of Wisconsin Virginia Tech
2

How often do we collect the following types of metrics around service health (effectiveness)?
80% 60%
45% 48% 34% 31% 30% 33% 59%

40%
20% 0%

25%
10% 10% 14% 7% 6% 9%

26%

11%

Demographics

Usage/demand

Performance

Customer Satisfaction

Weekly/Daily/Continuously Monthly or by Semester Annually Less frequently than annual or not at all
3

For what services do we collect metrics?

100% 80% 60% 40% 20% 0% Most of our services A few services Only key services All of our services 33% 29% 29% 10% 0% No metrics collected

Good news is that no one said zero!

And, our metrics to measure business efficiency and delivery of goals?

100% 80% 60% 40% 20% 0% Time (speed of delivery) Cost Quality (defect/error rates) Other

OTHER: 1) It widely varies depending on the service 2) We do not collect any business efficiency metrics 3) Project delivery # of calls abandoned; # change requests; # e-mails; # abandoned calls, resolution time, cycle time, abandonment, etc.; capacity, mean time to repair

Our use of targets

100%

80%
68% 63% 53% 32% 21% 20% 0%
Expectations Targets based Expectations Targets based Other (please We dont use based on our on specify) based on our on our peers any Service-Level- historical/trend customers performance demarcation of Agreements data requests/needs health

60% 40%

16%

OTHER: 1) Working on the use of ITIL Information Technology Infrastructure Library 2) Note: we don't do this consistently though 3) We do not use any service target range metrics 4) Industry Practices / Standards

Metrics are collected and analyzed primarily as

100% 80% 67% 60% 43% 40% 20% 0% 10%

5%

Grass roots effort Organizational/departmental effort Institutional effort Other (please specify)

OTHER: System performance metrics in transition to organizational effort.

7

Who is the audience for our metrics?

100% 80% 67% 60% 43% 95% 95%

40%
20% 0%
Internal IT staff IT management University executive leadership Your user community

38% 29%

Peers in other institutions (for benchmarking)

Other leadership

OTHER: Post publicly

How do we share them?

100%

80%
67%

60% 40% 20% 0%

53% 40% 33%

27%

Published for Other (please the organization specify) (Intranet)

Published publicly (web with open access)

Directly to Published for customers current and (electronic or potential hardcopy customers (web reports) with controlled access)

20% 81%

40%

60%

80%

100%

0% Made process/project adjustments

71%

Communicate better with our customers

71%

Communicate better with our leadership

Benefits so far

71%

Improvements

62%

Insights to the causes of problems or innovations

43%

OTHER: 1) right-sizing the organization; metrics enable us to tune documentation and training and better prepare support providers
Early-warning-system, enabling us to prevent problems 10% Other
10

How do we rate the maturity of our organizations use of metrics?

100% 80% 60% 40% 20% 10% 0% 0% Fully Mature 0% Maturing Managed Developing Novice 5% Totally novice
11

52% 33%

Our use of external data sources

100% 84%

80% 68%
60% 40%

72% 77% 56%

Provide data to Compare data to

28%

23% 8% 8% COFHE

20%
0%

16% 17% 11% Educause Core Data IPEDS

Use for defining metrics Don't use

OTHER: 1) Gartner for Benchmarking 2) Used to participate in the campus computing survey 3) Gartner

12

Any BI action?
100% 80%
60% 45% 40% 20% 0%

30%
15%

10%

5%

Have considered

Have not considered

In the process Have a of developing functioning BI environment

Other

OTHER: Currently considering an environment, platform selection pending

13

Our biggest challenges

100% 80% 60% 40% 20% 0% 81% 67% 43% 33% 33% 14% Lack of dedicated resources Lack of consistent comparison/benchm ark data Lack of automated collection tools Other (please specify)

5%
Lack of support from leadership

Lack of expertise in the development of metrics

OTHER: 1) Continuing engagement from mid-level leadership to respond to metrics findings 2) Organizations ability to identify specific KPI's to measure specific objectives 3) Changing leadership/definition of what is necessary and relevant; metrics must mean something to be used effectively; lack of a plan; staff resent 14

Lack of expertise in data analysis

What would we find useful?

0%
Standard definitions Guidelines on developing/reporting A template for basic metrics Review by peers of possible tools On-call expertise Other (please specify)

20%

40%

60%

80%

100% 90%

68% 63% 58% 37% 11%

OTHER: 1) None of the above 2) Unified approach to metrics from an organizational perspective; lack of a plan; dedicated resources would be better. No one is going to use another template and different services would be measured by different metrics unless the metrics were provided at a very very very high level
15

Tools what have we used, what do we think?

20

95%

100%

18
16 14 12 10 8 6 4 2 0
1 Excel 6 5 4 11

80%

60%

44%
1

41%
2

38%

40%

18% 13% 13% 7%

1 1 Other

20%

3 1 SAS

7%
1 Tableau

2
Cognos

1 SPSS

2
BSC

0%
iDashboard

0%
0%
Vision

Power Pivot SigmaXL

Inadequate

Fair

Good

Excellent

Outstanding

Used
16

Believe that the process and commitment to consistent data collection is far more important than the tool

Lessons learned
Metrics have helped to highlight areas of significant service difficulties (e.g., with BlackBerry services) and to note some low-level points of problems (e.g., around some of our network measures.) At the same time, our current metrics processes are highly manual in nature and require significant time investment to collect and report. We have seen challenges in getting service management engaged on the data writ large, which can lead to problems when errors due to service changes are missed thus impacting trending. Goals for us in coming year include focusing on trend analysis/reporting through executive summarization (done), gaining more mileage out of system-generated metrics on availability and low-level alarms, improving automated collection of non-availability data, and looking to focus data aggregation of human-generated, automated and other data into a dashboard to reduce effort level required to visualize service data. Benchmarking is very challenging because of the variance environments at each institution. Cost components may be different, service features and SLAs may not match, accounting practices can be problematic, tracking labor is different, etc.
17

Lessons Learned
We had a nascent metrics program under development with dedicated resources, focusing on helping service managers to develop metrics with their local data. With the departure of that resource in October, we are choosing to re-prioritize the work away from dedicated attention to metrics at this time. Instead, we watch with great interest the aggressive agenda that EDUCAUSE has developed with the reinvigoration of ECAR under Susan Grajeck. We will continue to monitor the progress of the various EDUCAUSE initiatives around research, data, and analytics and pursue collaboration opportunities based on our own priorities and resources. We did quite a push to get a metrics dashboard going a couple of years ago which was quite successful. However, the backend work of building a data metrics repository was never completed. This has limited us from getting deeper analytics questions answered and still requires us to perform manual queries often. On the other hand, when we recently needed to pull together a metrics dashboard for a large client (a hospital) we were able to reuse much of the work we had done previously.

18

Lessons Learned
We collect a lot of operational performance data using traditional tools (Cricket, Nagios, home-grown scripts) but don't have a reasonable dashboard or approach to making the data useful. We have recently started measuring performance of our service desk and groups behind them to track delivery against SLAs in our service catalog. We've started a Service-Now implementation and expect to use metrics delivered by that tool. Challenge getting consistent operational definitions both for internal use and benchmarking; Data collection is still a time consuming, manual process that we are working to automate through the collection of metrics from disparate systems into a BI environment; We are exploring the use of Microsoft BI tools (e.g. PowerPivot, SQL Server 2012, PowerView)

19

THANK YOU

20