Pragmatic Reporting for

DevOps Managers

Measure the DevOps KPI's that matter   

By
Roger Wallace
Director, Business Optimization,
ServiceClarity
Disclaimer:

This information within this e-book does not constitute or form part of, and should not be construed as advice, a recommendation, an offer, invitation, solicitation or
inducement to invest in ServiceClarity. All content belongs to ServiceClarity and is subject to copyright.
 
The material in this e-book has been prepared by ServiceClarity and is general background information about ServiceClarity. It is subject to change without notice. The
information in the e-book is given in summary form and does not purport to be complete and is for information and illustrative purposes only. The material contained in this e-
book may include information derived from publicly available sources that have not been independently verified.  
 
No representation or warranty is made as to the accuracy, completeness or reliability of the information in this e-book, including forecast financial information. Before acting on
any information you should consider the appropriateness of the information having regard to these matters and in particular, you should seek independent financial and/or
legal advice. To the maximum extent permitted by law, none of ServiceClarity, its directors, officers, employees or agents, nor any other person accepts any liability for any loss
arising from the use of this e-book or its contents, or otherwise arising out of, or in connection with it.

1 www.serviceclarity.com
 Contents Page

Introduction…………………………………………………………………………………………. 3

Planning KPI……………………………………………………………………………………….... 5

Throughput KPI..………………………………………………………………………………….. 7

Quality KPI…………………………………………………………………………………………… 8

Customer Satisfaction KPI……………………………………………………………………. 9

Conclusion………………………………………………………………………………………….... 11

2 www.serviceclarity.com
 Introduction to DevOps and Agile

Agile and Lean techniques are well established in the Software and Tech industries. They bring an
approach to software development that focuses on delivering quality products at speed, enabling
projects to iterate quickly and to respond to changing market conditions and user demand. Now
businesses are applying agile ideas across the entire delivery life-cycle from concept, through
development, to release and even on to customer success. DevOps is the application of agile methods to
prioritize above all, the continuous delivery of working software that meets the needs of the users. It
takes software from development to operations and beyond and enables businesses to innovate faster
and to adapt and above all deliver business value at high velocity.

However, the management of a DevOps project can be challenging as it requires more continuous
effort and more subtle attention than traditional project management. Before, project managers would
plan based on fixed specifications and known milestones, whereas now, DevOps projects embrace
change, working to deliver functionality to users in short bursts - adapting the deliverables to meet the
deadlines. Because DevOps applies to the entire lifecycle of a project we are forced to breakdown the
traditional silos of Development, QA, Operations and Support. We need to build teams that communicate
and work together to smooth the flow of ideas, using real-time feedback to deliver customer success.

Distributed Organizations and Teams

Alongside this shift in methodology, we are seeing a change in the formation of teams within modern
organizations. Today, it is common for project teams to include employees who don't operate out of the
same office, live in the same time zone or use the same development practices. And yet, despite these
factors, teams are expected to meet the needs of any given project within a specified timeline. When you
combine this with the short term iterative focus of agile, this can lead to difficulties in truly understand
the status of a project. All of these factors combined are leading management to seek out new reporting
tools that create strategic metrics. This is what is necessary to allow them to effectively manage in this
new dynamic environment.

The Product Owner

In the agile world, with project responsibility passing to the Product Owner, the PMO takes on a more
advisory and consultative role, rather than a controlling role. Alongside this, the PMO is responsible for
creating a well-trained workforce, ensuring that they are armed with the right processes and tools to
carry out their tasks effectively. They are expected to do all of this whilst keeping the project aligned to
the company’s overall goals.

3 www.serviceclarity.com
 Figure 1.1

ServiceClarity supports the integration of all four

elements within the development process:
Stakeholders, Development, Operations and the
Customer.

It is clear that the primary responsibility of the Product Owner now leans more towards delivering
business value, rather than simply managing project costs and schedules. This is a huge change for
businesses. More responsibility is placed on the operations side of the organization to provide project
direction, regardless of whether or not they are prepared to take on such responsibility. Putting in place
proper reporting and feedback processes with well-defined metrics, supported by the appropriate
software tools, are fundamental to the success of this new model.

Using Jira to Manage DevOps

In order to manage this dynamic agile environment, organizations across the world are implementing
DevOps tool stacks, which includes tools to facilitate team communication, coordination, reporting and
management. Atlassian’s JIRA is used by over 48,000 organizations to manage their agile software
development processes. It is one of the most widely deployed bug-tracking, issue-tracking and project-
management software tools in the world. In saying this, once the power of JIRA is understood,
administrators tend to become flooded with project creation requests. This can often lead to “JIRA
sprawl” which, in turn, can create problems for management who only need a top level view of status
across all their projects.

DevOps reporting with ServiceClarity is one way to give management the metrics they need,
whilst also enabling them to improve the efficiency of the entire process. DevOps reporting provides a
feedback loop that is the key to understanding how well teams deliver, how fast processes change and
how best to help people plan when a company switches to agile. Another advantage of this type of
reporting is that it also seeks to be inclusive, bridging the gaps between teams outside of the development
organization. This enables stakeholders and teams, outside of the loop of development tools such as JIRA,
to stay informed. They are then able to plan based on real-time feedback, therefore reducing the ‘Silo’
effect, which can often lead to delays and cost overruns. Even in an agile world, businesses still need to
answers the questions: When will we be done? and Do we have the capacity to deliver?

4 www.serviceclarity.com
 Metrics for DevOps Reporting

1. Planning: Remaining Story Points

Agile was conceived to help practitioners develop new ideas, create new products quickly, deal with
changing deliverables and manage unexpected problems. Constantly changing requirements are often
cited as a fundamental aspect of IT delivery and, to some extent, an organization that has adopted agile
methods will have accepted the inherent difficulties involved in planning and predicting these complex
processes. Unfortunately, this unpredictability is only one of the many factors that can affect release
dates. We have witnessed that, in the long term, a team’s understanding grows over time and this tends to
result in accelerated project delivery. However, more often than not, this growth in understanding also
exposes flaws in the estimates and exposes problems that were not anticipated.

We see adopting DevOps as an opportunity to optimize the speed at which you can deliver but
also help keep work in progress (WIP) to a minimum, sustainable level, whilst also continuously
maintaining quality and adapting to change. A DevOps project seeks to limit exposure to change by
planning short, achievable milestones. Often, the work in progress is deliberately limited to one or two
week “sprints" but when our attention is completely focused on ‘now’ we can lose focus on the more long
term goals. This sprint approach has wide reaching implications and impacts the organization outside of
the development and operations teams.

Management is also aware that, outside of the DevOps project, there are stakeholders that need
accurate projections in order to integrate their work and plan their deliverables. For example, the
marketing team cannot plan for a product release when they don't quite know when it will be ready, or
what features it might offer. This lack of foresight effects customer success teams as they are unable to
train their support staff for a product that they yet don’t have. Lastly, and above all, customers will always
want to know when new features will be ready or when their issue will be fixed.

Therefore, teams that adopt DevOps and adapt to the agile ‘short achievable sprints’ method still
need to plan for delivery to the customer. This is where we look at the wider organization when adopting
DevOps delivery. It is imperative that expectations are set, progress is tracked and that slippage is spotted
early and mitigated. Tracking these metrics will easily help you answer the questions: What have we got
left to do? When will we be done? Will we hit the planned release date?

www.serviceclarity.com
5
 Managing the work in progress, the work left to complete and how these work packages move
between teams is what tools like Atlassian's JIRA are made for. JIRA is a valuable tool, essential for the
day to day tracking of work in an agile project. However, long term planning and coordinating across
projects or teams involves stepping back from the day to day and looking at the bigger picture, tracking
what has happened but also projecting for the future.

ServiceClarity fully supports JIRA's query language and can track in real-time the number of
story points in outstanding JIRA tasks in one or more projects, in Epics, the 'current sprint' or any other
custom query. It will also track backlog by team, component and release. With such knowledge of the
WIP and the work remaining, management can understand how far the team have progressed and how
much they have left to go. Providing, in real-time, invaluable feedback on when features and fixes will be
ready and enabling coordination of other teams.

Figure 2.1
ServiceClarity Remaining Story Points Report

6 www.serviceclarity.com
 2. Throughput: Story points completed this week/month

DevOps offers a methodology to create quality output at speed and, with that, the ability to respond
to customer demand and competitive pressure. Fundamentally, by adopting a DevOps approach as an
organization you seek to add value by delivering a working product with new features and fixes in
quick iterative steps. By reducing the lead time from concept to production, an organization can
experiment, interact with users, and tighten the feedback loop.

Whether you are just beginning to introduce DevOps into your organization, or you are
already on a continuous improvement plan to optimize your existing process, it is important to
understand the throughput of you teams. The throughput of a team or project is the amount of work in
progress (WIP) that is put into production in a given period, typically measured over a calendar month.
If you have high throughput, you are not only releasing a lot of fixes, improvements or features into
production in any given period, you can also start many new initiatives or projects without increasing
WIP. A low WIP and high throughput gets you faster feedback, so you can adjust to the actual demand
of the market.

A question often on management’s mind is, if I change a process or a team structure how will it
affect productivity? One of the key measures for this, which also indicates a successful DevOps team,
is to understand ‘how fast are we going’. Tracking the throughput of your DevOps team enables you to
answer practical questions such as: Did increasing the team size increase our velocity or slow us
down? It also helps the wider business understand the value of the DevOps delivery. The DevOps
research & assessment (DORA) organization’s study on "Forecasting the Value of DevOps
Transformation”(1) introduces the concept of DevOps as a revenue generating process, formulating a
measure of the impact of DevOps based on the speed at which new ideas, product features and
product lines are introduced. However, to demonstrate the value of DevOps, as well as optimize the
process, you need to track your team's throughput.

Team throughput is also a key metric that enables more accurate estimates of long term
planning. Tracking throughput in real-time can provide the feedback necessary to adjust and plan the
next sprint and predict the ultimate delivery date.Task tracking and management tools, like JIRA, can
support many different ways of tracking and estimating effort, including story points. ServiceClarity
goes one step further by tracking the total number of JIRA story points completed on a daily basis and
reports on these weekly, monthly and yearly. ServiceClarity is specifically designed to aid management
with future planning, helping them determine the success of changes and communicating the value of
DevOps to the broader organization.

7 www.serviceclarity.com
 3. Quality: Percentage time spent on unplanned tasks

The two pillars of DevOps, quality and speed, are in constant tension. As you optimize your process
there is a need for balance between blind, relentless speed, and methodical, gated testing and quality
assurance. In a DevOps organization, automation is a vital tool for improving reliability and trust, whilst
also maintaining speed. But how do we measure the quality of our output?

Just as "lines of code" are reductive and unhelpful when measuring the output of a software
development team, it is important to measure the quality of DevOps output in a way that tracks
progress without the assumed finger pointing that quality metrics often imply. This is especially true in
organizations that rely on multiple teams to deliver. ‘Blame metrics’ are well known to encourage
gaming of the system and optimization for the wrong thing. They do not increase cooperation and
communication.

In a recent 2017 study on "The State of DevOps” (2), commissioned by Puppet Labs, an analysis
of over 20,000 surveyed IT professionals concluded: "This year, we are happy to report that high-
performing organizations spend 21 percent less time on unplanned work and rework, and 44 percent
more time on new work."

If we examine the underlying cause and effect of this finding, we can clearly see that poor quality
work leads to rework, and time spent working on fixing problems in production is strongly correlated
with lost productivity. More than that, we should be clear that racing to hit the current deadlines will
impact the next delivery and then the proceeding deliveries after that. IT professionals at the coal face
know this to be true and tracking the amount of rework or unplanned work in a DevOps delivery is a
measure of product quality. Therefore, it does not blame but, instead, shines a light on issues that the
teams are probably already aware of and are willing to work on. Most importantly, it also enables the
organization to optimize the right thing, the time spent on new work, and on adding value to the bottom
line.

The tracking of this unplanned work can readily be achieved in a tool like JIRA. Further to this,
ServiceClarity can also track the time spent on each JIRA task and can calculate the percentage time
spent on bugs, support and other unplanned work. Calculations can be made daily, and reported upon
weekly and monthly. Fundamentally, this allows you to optimize your product quality whilst minimizing
time wasted.

8 www.serviceclarity.com
4. Customer Satisfaction: Percentage reopened tickets

The ultimate measure of success for any IT delivery is the satisfaction of the users. This is true regardless of
the process used to deliver. Whether or not DevOps or agile is practiced, there is a need to understand if
what you are delivering is meeting your customer’s needs. Satisfaction is a notoriously difficult quantity to
measure and there are many proxy metrics promoted to help understand how an organization, product or
service is performing in the eyes of its users. One challenge with measuring customer satisfaction is the
subjective nature of the problem and the commonly proposed solutions. For example, Net Promoter Score
(NPR) is a prominent marketing metric that attempts to gauge customer loyalty through questionnaires.
Unlike most questionnaire based solutions, NPR reduces to a single number between -100 and +100 that
should indicate how likely a customer is to recommend your product or service.

From a DevOps perspective, it is important to capitalize on your increased delivery speed and
agility. However, an organization’s ability to react to customer demand is predicated on its capacity to track
customer satisfaction. Despite its popularity, NPR still offers subjective and delayed feedback. This raises
the question: what practical measurements can we make that can help inform and optimize the IT
development process in real-time?

One possible proxy for customer satisfaction might be the total number of support issues raised or
bugs reported. Although this is a valuable metric that should be monitored, it speaks more to the quality or
usability of a release than to the satisfaction of your customers. Many issues will be duplicates, many will be
low impact, many more will simply be issues with education or expectation.

The fact that a customer raises an issue or contacts support is not necessarily a sign of
dissatisfaction. A successful interaction with your customer support team should leave a customer satisfied
that their question has been answered, their concern has been heard or that their issue will soon be
addressed. Handling customer support requests is an opportunity to impress and to take on feedback.
When asking if your customers are satisfied you should ask: are we getting support right first time?

ServiceClarity will track your issues in JIRA and, by simply including a process stage in your JIRA
workflow that acknowledges that a once resolved issue has been reopened, you can use ServiceClarity to
track work that doesn't get resolved first time. ServiceClarity can then compare the number of issues
against the total volume and thus enables you to track percentage reopened, or your percentage right first
time.

9 www.serviceclarity.com
 Figure 3.1
ServiceClarity DevOps Report

10 www.serviceclarity.com
 Conclusion

The current business environment is one in which disruptive technology challenges established business at
every turn. To survive businesses must deliver more value and innovation in less time. Agile DevOps helps
achieve this by delivering quality software more regularly, reliably and predictably. Software-driven
innovation separates successful businesses from failures, and while Agile DevOps enables technical
innovation, stakeholders setting business strategy need this technical innovation translated into business
metrics stripped of DevOps jargon. ServiceClarity bridges the gap between technical innovation and
business innovation by reporting the value of DevOps in Business Terms. ServiceClarity applies Agile
principles to Strategic Reporting by delivering management reports early and often, prioritizing the right
business metrics and improving communication between business functions.

To disrupt, rather than be disrupted, find out how ServiceClarity drives software-driven innovation
in successful businesses. For more information on ServiceClarity and the service we offer, please visit
www.serviceclarity.com or contact one of our team on info@serviceclarity.com.

11 www.serviceclarity.com
 References

(1) DevOps Research and Assessment (March 2017). Forecasting the Value of DevOps Transformation,
Measuring ROI of DevOps. Available at: https://devops-research.com/roi

(2) Puppet Labs & DevOps Research and Assessment (2017). The State of DevOps Report 2017. Available
here: https://puppet.com/resources/whitepaper/state-of-devops-report

12 www.serviceclarity.com
 Thank you for reading our e-book. If you have
any queries about our product, feel free to
contact us at any time.

Click Here to enquire about Automating your

DevOps Reporting with ServiceClarity

Contact Information:
w: ww.serviceclarity.com
t: +44 (028) 90 236 742
e: info@serviceclarity.com
a: 10B Weavers Court, Business Park, Belfast, BT12 5GH