2019

Through the SIG

looking glass

// Looking back on 10 years of software analytics

// Sharing lessons for the future
Table of Contents Foreword

� Foreword
� Software quality
� Research questions
� Key findings and recommendations
� Has software quality improved or declined?
� Do better software processes produce
3 At SIG, we have been helping our clients improve their software health
since we founded the company in 2000. Nearly 20 years have passed since
4
then, full of evolution and progress in the software industry. Today, we
11 have the ability to look back and reflect on the data that we have collected
along the way. What do those numbers tell us now? Has the industry
seen improvements across the board? What differences do we see in the
11 software technologies that are emerging and declining? What practices
separate the low from the high performers?

better quality software? 20 In this report, we present our view through SIG’s looking glass; our state of
� What’s the future impact of emerging the union address, as it were, which kicked off the 2019 SIG Symposium on
September 26 in Amsterdam. Our goal is to provide you with the relevant
technology stacks? 24
insights you need to improve software health wherever you go next.
� Future software analytics 32
� Final thoughts 33

Dr. Luc Brandts Dr. Magiel Bruntink

Luc is CEO of Software Magiel is head of Research at

Improvement Group. He is a
seasoned IT professional having run
and managed several technology
firms. Luc co-founded BWise in
1994, growing it to a leading global
technology firm.
Software Improvement Group. He is
an internationally published author
in the field of software engineering,
with over 15 years of experience in
research, consulting, and education.

2 3
Software Quality

Software quality is one of those things whose importance nobody

QUOTE-RIGHT “High quality software is cheaper to produce.”

“When thinking about internal quality, I

would deny. In the same vein as road traffic, health care, and
stress that we should only approach it as an
education, quality is pertinent to the software industry. The devil
economic argument. High internal quality
is, however, in the details of the definition, and often the cost that
reduces the cost of future features, meaning
people are prepared to pay to reach it. It’s therefore a good thing
that putting the time into writing good code
that the software industry has arrived at various standards to help
actually reduces cost.”
streamline this discussion.
Martin Fowler, Chief Scientist,
SIG is a strong proponent of applying the ISO/IEC 25010 standard to
ThoughtWorks, 2019
provide a stable definition of the quality of software products. The
standard drills the overall topic down into sub-components such as
Performance Efficiency, Reliability, Security, and Maintainability. It then tells
us what aspects matter further down. For instance, to have a Maintainable
SOFTWARE QUALITY IS
software product, it needs to be Modifiable and Testable. A KEY SUCCESS FACTOR
Approximately 12 years ago, SIG developed a measurement model that “Organizations that elevate code
takes over where the ISO 25010 standard leaves off [1]. SIG’s model for maintainability provide real advantages to
Maintainability provides concrete source code metrics to make the ISO their engineers.”

QUOTE-RIGHT
standard tangible. Furthermore, by comparing these code metrics to
a database of thousands of software systems, SIG provides a unique “Investments in code maintainability, loosely
benchmark of software quality. coupled architecture, and monitoring help
support SDO performance (via continuous
SIG’s measurement model for Maintainability is the only model of its kind delivery) and productivity (via reductions in
that has been approved by TÜViT for the use of quality measurement and technical debt), highlighting the importance of
formal certification of software products [2]. good tooling and systems.”

Accelerate State of DevOps,

DORA, Google Cloud, 2019
[1] Ilja Heitlager, Tobias Kuipers, and Joost Visser,
"A practical model for measuring maintainability”, QUATIC 2007), IEEE, 2007.
[2] https://www.softwareimprovementgroup.com/wp-content/uploads/2018/05/20180509-SIG-
TUViT-Evaluation-Criteria-Trusted-Product-Maintainability.pdf

4 5
SIG applies the ISO/IEC 25010 SIG has its own Maintainability
standard for software product model. Approved by TÜViT
quality

Functional 1. Measurements
Suitability

Aggregate
Performance measurements to
Security quality profiles

2. Quality profiles
`
Compatibility Maintainability
Translate quality
profiles to scores
for system
characteristics HHHII
Usability Portability
HHHHI
3. System-characteristics
HHHII
Translate scores
for system +5 …
characteristics
Reliability to scores for
ISO 25010 sub HHHHI
characteristics
HHIII
4. Sub characteristics
HHHII
Maintainability serves as a strong basis for analysis +2 …
Translate to
overall rating of
technical quality
(Maintainability)
Initiation Build Acceptance Maintenance
5. Overall rating

20% of the c osts HHHHI

80% of the c osts

6 7
 5

4.5

SIG Monitoring increases

4

SIG Software Consultancy 3.5

Software Quality and prevents

3

2.5 decline
2
Nowadays, there is no lack of software measurement tooling in
the marketplace. Many mature software development teams apply 1.5

some form of automated quality control. However, this is often not HHHHH
5
1 MONITOR
sufficient to improve or even maintain quality. Tools are imperfect, 4.5
provide false positive results, or fail to prioritize their outcomes by
HHHHI
urgency and feasibility [3]. 4
3,7

MAINTAINABILITY
3.5

To tackle this issue, SIG’s core service is software consultancy, which HHHII Exclamation-Triangle
3
employs automated tools and measurement models in combination with
expert consultancy on the outcomes and the implications. As software 2.5 2,6 2,7
consultants, we interact with our clients at many levels, from executive HHIII
2
and management to operations and development, in order to align on the
1.5
software quality that our tooling has objectively measured.
HIIII
1

In a software monitor project, the source code of a system or application is 2012 2013 2014 2015 2016 2017

measured by SIG in a set frequency. Often the frequency is weekly, allowing

a high resolution timeline to observe quality levels and trends. Each month
or quarter, our consulting teams aggregate the recent measurements to Exclamation-Triangle The drop in technical quality significantly
provide recommendations and advice. increases maintenance costs
Decrease in technical quality: 1.0 H
SIG’s data collection consists in part of the measurements done in software Increase in maintenance costs: € 290,000
monitoring projects, of which several hundred are active each year. Period: 3 years

[3] Maria Christakis, Christian Bird, This calculation is based on system quality, size, technology stack and expected future
“What developers want and need from program analysis: an empirical study”, ASE, 2016. modifications to the system. The result is cumulative. This graph contains real data, but is
anonymized.

8 9
 SIG
Software Consulting Research Questions
by the Numbers
In this 2019 report, we provide you with our insights from a data

250+ set spanning ten years, containing between 3,000-4,000 systems of

which SIG analyzed the source code.

450+ technologies from

mainframe to mobile In addition, we combine the source code measurements with outcomes
customers from a software process survey our consultants have been running over

25+ billion
the past five years. You’ll find that these data insights nicely complement
market analyst reports such as ‘Accelerate State of DevOps 2019’ by DORA
and Google Cloud.
lines of code in analysis
warehouse This year, we will focus on the following three research
questions:

1. Has software quality improved or declined?

In the grand perspective of the software industry, we feel overall
insight is missing on whether software quality is actually improving or
whether it’s in decline.

2. Do better software processes produce better quality

software?

100+ million
Commonly, executives and practitioners alike expect the quality of

187,000+ lines of code analyzed weekly

software processes to have a large influence on the quality of the
resultant products. We are in the position to correlate data sets on

system snapshots both these angles, and thus can address this expectation head on.

3. What’s the future impact of emerging technology stacks?

4,000+ Reflections on the past are not complete without projections into the
future. What do we at SIG see emerging in our recently collected data,
systems evaluated and what does that mean for decision makers in the field?

10 11
What are the key findings
in software trends?
1. Has software quality improved or declined?

Since around 2015, we are seeing subtle but sure year-over-year

improvements in the averages of our software quality metrics. In January
2015, we saw an average maintainability rating of 3.1 stars for the
approximately 100 systems measured that month. Then in January 2019,
we see 3.4 stars on average based on about 700 systems measured. Chevron-RightChevron-Right DID ISO 25010 Maintainability, on average, improve over the years?
MONTHLY AVERAGE MAINTAINABILITY

12 13
Chevron-RightChevron-Right There are several trends underlying this overall positive quality vibe. First,
on average, the systems that we see have a smaller volume of code. As
a core factor in maintainability and cost, an overall reduction in volume
is very positive. Smaller systems are easier to maintain, and if necessary,
cheaper to replace. Chevron-RightChevron-Right

How did the average system volume evolve?

MONTHLY AVERAGE SYSTEM VOLUME (PERSON MONTHS)

14 15
Chevron-RightChevron-Right System technology (or tech stack) is an important factor. Systems in the
‘Low code & BPM’ category are leading improvements, allowing for small,
high quality systems that have lower expected spending. On the other
hand, systems in ‘Legacy/3GL/4GL’ instead see a dwindling trend in quality,
remaining at below average quality levels.

Do different technology stacks show different maintainability trends?

MAINTAINABILITY

16 17
 Recommendations
‘Has software quality improved or declined?’

Easier said than done, but our numbers are clear. Migrate out of
the Legacy/3GL/4GL category as soon as it’s feasible. Remaining
with legacy technology will further harm the speed at which your
systems can evolve in the future. They will require high upkeep
costs as long as they remain operational.

Exclamation-Triangle We recommend creating a clear picture of your software portfolio showing

the amount of code still stuck in the Legacy category, and then short-track
that to modern software technology.

SIG is a strong believer in measurement. “To measure is to know,” to

paraphrase Lord Kelvin’s famous statement. What we see is that systems
that are monitored consistently for their software quality, in at least a
weekly to monthly frequency, have a far smaller chance of slipping towards
lower levels of quality.

Exclamation-Triangle We recommend consulting your IT leaders today on what measurement

programs they are currently running and stress the inclusion of objective
quality measurements to prevent further quality decay.

18 19
2. Do better software processes produce better
quality software?
Do better software processes produce better quality software? It really
depends. We observe the key correlations in our multi-year survey on
Development (Best) Practices Assessment, or DPA.
In most categories, the adoption of development best practices has
increased over the past 4 to 5 years. Most pronounced are the increased
AVERAGE DPA THEME SCORE
20
adoption of automated tools for ‘Coding and Building’ and ‘Deployment.’
The greater availability and accessibility of continuous integration and
deployment tools (e.g. DevOps) are at the heart of this trend.
Drilling down in the data set, we pitted the low performers at producing
maintainable code versus the high performers. We then see a stark difference
in particular for the development practices that can be automated: Chevron-RightChevron-Right
Did use of development best practices increase over the years?
21
Chevron-RightChevron-Right 61% of teams that produced 4 or 5 star software, use full continuous
Recommendations
integration, while for 1 or 2 star software, we only see 18% full adoption. ‘Do better software processes produce better
Other automated practices such as testing (53% high vs. only 3% low) or quality software?’
deployment (58% high vs. 22% low) see similar rifts in adoption levels.

Automation in development is a clear benefit in terms of

efficiency, overall consistency, and quality of work. Developers
like to automate, not only implementing functionality, but also to
improve their own processes.

Teams that automate with a certain zeal seem to have the edge, as they
keep their hands and minds free of the more mundane and repetitive tasks,
allowing focus on the more creative and challenging mental tasks.

Exclamation-Triangle We recommend defining at least the following key questions to address in

your IT organization:

„„ What key obstacles exist that hinder further development

automation in your organization?
„„ Are quality control tools part of your continuous integration
pipeline? If yes, what actions do you take on deviations?
„„ What is the mean time elapsed between the decisions to release
functionality and their actual deployment in production?
„„ What is the proportion of defects found during (automated)
integration and testing versus defects found in production?

22 23
3. What’s the future impact of emerging Historical trends & forecasts of language popularity
as analyzed by SIG
technology stacks?

In recent years, SIG has witnessed a surge in systems and

applications built on relatively modern technology stacks, together
with an overall drop in the average application’s code volumes.
Both these trends are reasons to smile, as benefits like reduced
maintenance costs and higher agility remain in sight.

We saw before that applications built using Low Code and/or BPM
technology show a strongly positive quality. This category contains
technology such as Mendix, Outsystems, Blueriq, and BPEL. Also
applications in Scripting and Mobile technology tend to perform above
average with respect to code quality.

To see what’s emerging we keep track of (peak) popularity of each

programming language in our database of measurements. Currently
peaking, and forecasted to further grow are languages like Java, JavaScript,
and Mendix. Languages that recently peaked but are likely to become more
popular are C#, TypeScript, and Python. Then we also see languages that
are well beyond their peak: C++, PL/SQL, and COBOL.

24 25
Historical trends & forecasts of language popularity Historical trends & forecasts of language popularity
as analyzed by SIG as analyzed by SIG

26 27
Recommendations
‘What’s the future impact of emerging technology stacks?’
Employing modern technologies like those in the categories JVM
(Java, Groovy, Scala, and others), Microsoft (C#, .NET), and Low
Code & BPM, today seems like a safe bet. However, are there
reasons to be careful? Yes, indeed.
Low Code and BPM are doing well on most metrics, but watch
out for architectural quality.
28
In particular for the Low Code & BPM category, we observe a trend
that also warrants caution. Most code properties there trend upwards,
except for two properties that consider architectural quality: Component
Independence and Module Coupling. Component Independence checks if
a system is built from independent components that can be developed in
parallel. Module Coupling measures the internal coupling in a system, or
the degree to which there is spaghetti code.
29
Applications are at risk of becoming too strongly connected to their low Exclamation-Triangle However, strict control of application quality and code volume is amongst
code platform, and/or suffer a suboptimal architectures due to platform- the most important measures. Apps and components should remain
dictated choices. This could yet cause them to slide down the quality slope. replaceable on a timescale of months rather than years.
How to prevent these now-shiny applications from becoming tomorrow’s
legacy? Generic advice in software architecture is tricky as a lot depends on
the domain and the existing portfolio.

30 31
Future Software Analytics Final Thoughts

SIG is active in several academic-industry research programs that
will help shape the future of software measurement and analytics.
In the FASTEN project, funded by SIG and
the European Union, SIG is helping to
develop Intelligent (Software) Package
Management at the scale of the major
global software ecosystems. With that infrastructure, engineers will be able
to analyze whether software projects are using the best software packages
available, and if not, how to get there.
https://fasten-project.eu
Looking back over the past ten years, we are left with a feeling of
cautious optimism. There are signs that we, that is, the software
industry, are changing our quality game for the better. We see the
trends in various underlying metrics, on the code level, on process
level, and in the emergence of modern technology stacks that
enable more flexible and higher quality software.
Is software quality then mostly a concern in large legacy software systems
that were built decades ago? Yes, it definitely is still an issue there; not
only because the number of legacy systems and applications are growing,
but also because the legacy systems still remaining continue to grow in
code volume and technical debt. The only path forward is to migrate to the
greener pastures offered by more modern technology.

i-CAVE is a national collaboration on technological research for

However, we also see some signs that caution us on the modern
autonomous driving and cooperative vehicles. SIG is doing research on
technologies, which appear to produce more, finer grained applications,
safety and security concerns in software architectures, creating the next-
components, and services. The trade-off is that these modern architectural
generation architectural models that will allow for objective measurement
styles also result in higher levels of coupling, more dependencies, and thus
and higher quality software development in automotive.
call for stricter quality control at that level.
https://i-cave.nl/

Codefeedr is a collaboration between SIG and the TU Delft, co-funded by

NWO, focused on creating the next generation software data streaming
infrastructure. The aim is to provide developers and analysts with the
capabilities to see, aggregate, and query the actions of millions of software
development processes, globally, in near-real time.

http://codefeedr.org

32 33
In 2016, SIG published two guides for
practitioners on two of the core issues we
encounter in our software consultancy
business.

Building Maintainable Software (O’Reilly,

2016) on the essentials on software
maintainability explained with a focus on
practical application, rationale, and concrete
measurement.

Building Software Teams (O’Reilly, 2016) on

how to effectively employ the wealth
of software development automation tools
within software teams, to enhance their
practices such as continuous integration,
testing, quality measurement, and
deployment.
Colophon
In our view, the lessons taught in those guides are still starting points for
budding software teams today. Copyright © 2019
by Software Improvement Group (SIG)
Authors: Dr. Luc Brandts and Dr. Magiel Bruntink
At the same time, we are looking forward to upcoming trends in software,
such as a growing dependence on artificially intelligent systems, The information in this document may not be copied
opportunities and threats coming from new (open source) software or published, distributed or reproduced in any way
ecosystems, and new insights into the socio-technical domain of teams and whatsoever without the prior written consent of SIG or
organizational architecture. the legal consent of the owner.

Design: Plushommes
There is a lot to look forward to in the years to come! Art: Rawpixel

34 35
Fred. Roeskestraat 115 www.softwareimprovementgroup.com

1076 EE Amsterdam marketing@softwareimprovementgroup.com

The Netherlands

Ge tti ng so ftware r i g ht for a h ea l th i er d i g i t al wo rld