FOCUS: ACTIONABLE ANALYTICS
How Robust
Is Your
Development
Team?
Lu Xiao, Zhongyuan Yu, Bohong Chen, and Xiao Wang, Stevens
Institute of Technology
// A proposed automatic approach intuitively
visualizes development team hierarchy,
quantifies overall team robustness, and
identifies the point of risk for team robustness.
An investigation of six Apache open source
projects has shown its effectiveness. //
THE DEVELOPMENT AND mainte-
nance of a software system is a col-
laborative activity. In particular, a
complex, large-scale software sys-
tem requires hundreds of developers
contributing simultaneously, add-
ing new features, testing, and fixing
bugs. A robust development team is
a necessity for the success and on-
time delivery of the software product
being built.1 In the event that devel-
opers temporarily or permanently
64 I E E E S O F T WA R E | PUBLI S HED BY THE IEEE COMPUTER SO CIE T Y
leave the team due to various rea-
sons, such as job hopping, retire-
ment, and sick leave, 2 how much
could this change potentially disrupt
the rest of the team? Ideally, the rest
of the team should be able to quickly
adjust and make a smooth transi-
tion. Team robustness is critical for
overall project progress in a com-
mercial environment—the overall
progress hopefully won’t be substan-
tially dragged down by team change,
and the rest of the team can still
stick to the original project sched-
ule. It is equally, if not more, impor-
tant in an open source environment
because of the higher likelihood of
team change—contribution to open
source projects is purely voluntary.
In this article, we offer an auto-
matic approach to evaluate team
robustness based on social network
analysis. 3 Our approach is com-
posed of three major steps:
1. mining a project repository to
construct a weighted collabora-
tion graph (WCG),
2. developing a novel Two-Layer
Collaboration Hierarchy (TLCH)
to analyze the collaboration
structure hierarchy, and
3. evaluating team robustness by
estimating the information loss
associated with team members.
We investigated six Apache open
source projects, and the results show
that our approach greatly promotes
the understanding of development
team robustness and successfully
quantifies the robustness.
Our Approach
Our approach to evaluating develop-
ment team robustness is composed
of the following three parts.
Mining the Weighted
Collaboration Graph
First, we mine the project devel-
oper team mailing list to construct
a WCG. In a WCG, each node rep-
resents an individual developer,
weighted by the total number of
email communications the developer
participated in. A node with a high
weight indicates a developer who
carries a significant share of knowl-
edge of the system. If such a devel-
oper leaves the team, it is likely to
0 74 0 -74 5 9 / 1 8 / $ 3 3 . 0 0 © 2 0 1 8 I E E E
significantly affect the rest of the team. An edge between
two nodes is weighted by the number of shared email
threads between the two associated developers. The edge
weight implies the intensity of collaboration between
the developers. In a high-intensity-collaboration envi-
ronment, if a core developer leaves, other developers are
highly likely to be impacted directly.
Calculating and Visualizing the Collaboration Hierarchy
Next, we analyze the structure of the WCG. The goal is
to understand the hierarchy (if one exists) of the devel-
opment team. For example, if a single developer has a
significant amount of collaboration with others, we con-
sider that he or she is in a higher level of the hierarchy
than less-involved developers.
We developed an algorithm to cluster a WCG into a
simple TLCH. As the name suggests, the TLCH sepa-
rates the developers into two layers: the inner layer and
the outer layer. The rationale of the choice of two lay-
ers is to distinguish core developers in a team from the
others. We hypothesize that these core developers are
the point of risk for the robustness of the entire develop-
ment team. If they become unavailable, the overall proj-
ect development will be significantly disrupted. We will
provide measurements for estimating the robustness of a
team using the TLCH later.
The inner layer contains nodes whose weight is at
least one standard deviation above the mean weight of
the nodes in the WCG. The other nodes are aggregated
into the outer layer. If developers in the inner layer leave
the development team, the remaining team will be less
stable compared to if someone in the outer layer leaves.
One of the common outlier detection methods—a mod-
ified z-score (based on the median absolute deviation)4 —
serves as the criterion for a TLCH. If no outlier is de-
tected, the development team cannot be constructed as
a TLCH; therefore, there will be only one layer in the
WCG, which suggests that the team has a “flat” collabo-
ration property. If one or more than one outliers are de-
tected for the project, a TLCH will be applied. Teams
without a TLCH are considered to be more robust than
teams with a TLCH.
In a constructed TLCH, two layers are laid out in two
inclusive circles; both the node size and edge thickness
indicate the respective weight. The TLCH structure is vi-
sualized in Figure 1 and discussed in more detail later.
Admittedly, “one standard deviation different” is not the
only feasible choice to distinguish core developers from
others. Project insiders may have better knowledge to
tune this for more meaningful results in the TLCH. We
will discuss this as a limitation later.
Estimating the Information Loss Associated
with Developers
To quantitatively evaluate the robustness of a develop-
ment team, we estimate the information loss associated
with each developer, especially those developers in the
inner layer of the TLCH.
The information loss associated with a developer,
IL dev, is calculated as the percentage of email threads
that developer participated in. Intuitively, it reflects the
mass of information the developer possesses regarding
this project. Therefore, we use it to measure the percent-
age of information that will potentially get lost with the
absence of the developer. The larger the value for IL dev,
the higher the potential disruptions to normal operation
of the team caused by that developer’s absence. Formally,
WCG.dev ( ) .weight ( )
ILdev = N
. (1)
∑WCG.dev ( ) .weight ( )
i
i =1
To further estimate the overall robustness of a team,
we calculate the core information loss (CoreIL) as the
average information loss of all the inner-layer developers:
1  
CoreIL = ln  ∑
Inner _ Layer  i ∈ Inner _ Layer
ILdevi  .  (2)

CoreIL measures the potential information loss along
with the absence of any developer in the inner layer of
the TLCH. The higher this value, the higher the disrup-
tion in the event of an inner-layer developer’s absence,
and thus the less robust the development team will be.
Only the inner-layer developers are considered in the
overall robustness measurement, because these develop-
ers are the point of risk for team robustness.
Case Study Results
We studied six open source projects using this approach.
Case Study Subjects
The study includes six Apache open source projects, in-
cluding Cassandra, Hadoop, PDFBox, CXF, HBase, and
Camel. The basic information for each subject is shown
in Table I. It shows the covered length of history in this
study, the total number of developers, and the total num-
ber of email communications.
J A N U A R Y/ F E B R U A R Y 2 0 1 8 | I E E E S O F T WA R E 65
 FOCUS: ACTIONABLE ANALYTICS

(a) (b)

(c) (d)

(e) (f)

FIGURE 1. Development team hierarchy. (a) Hadoop. (b) Cassandra. (c) CXF. (d) PDFBox. (e) Camel. (f) HBase.

66 I E E E S O F T WA R E | W W W. C O M P U T E R . O R G / S O F T W A R E | @ I E E E S O F T WA R E
Collaboration Hierarchy Visualization
Figure 1 shows the TLCH calculated
from each project. (For the sake of
readability, each graph only shows
up to the top 20 most contributing
developers from each project.) In this
article, the TLCH for each project is
calculated as a static view covering
the selected period shown in Table I.
Applying the TLCH approach, all,
except Hadoop, contain the two
hierarchical layers.
Each calculated TLCH contains
two (CXF and PDFBox) to four
(Cassandra) developers in the inner
layer. These few developers have a
significant amount of collaboration
among themselves and usually (ex-
cept for PDFBox) are highly con-
nected with other developers in the
team. Thus, they play critical roles
in maintaining the stability of team
collaboration due to their significant
share of knowledge of the system.
If they leave, the team is subject to
severe information loss. Therefore,
these inner-layer developers are the
point of risk for the robustness of
each team.
In comparison, Hadoop shows a
flat team structure. This is because
the Hadoop team does not meet the
criteria of the TLCH algorithm. In-
tuitively, this implies that the collab-
oration among developers is evenly
distributed among all the team mem-
bers. We believe that a flat structure
is more robust than the hierarchy
structure because no matter who
leaves the team, only a small propor-
tion of the project knowledge will be
lost. Hence, the whole team is more
resilient to risk.
Information Loss
Figure 2 shows the information loss
trend along with the top four de-
velopers in each project. Since the
developers in the inner layer of the
Table 1. Subject projects.
Length of history—mm/yy
Subject (no. of mos.)
Cassandra 09/09 to 12/16 (87)
Hadoop 08/09 to 12/16 (88)
PDFBox 08/09 to 02/17 (90)
CXF 12/07 to 02/17 (110)
HBase 12/09 to 02/17 (86)
Camel 07/08 to 12/16 (101)
TLCH are likely to be the point of
risk for team robustness, we cal-
culate the CoreIL associated with
these developers. Inner-layer devel-
opers are highlighted as the thicker
dots in Figures 2b to 2f. The CoreIL
in the five projects with a hierarchi-
cal structure is between 0.346 (CXF
in Figure 2c) and 0.473 (PDFBox in
Figure 2d). This indicates that if any
of the inner-layer developers become
unavailable, around 34.6% to 47.3%
of the project information might be
lost. In particular, PDFBox has the
least-robust team according to the
data, whereas the CoreIL reaches
0.473. And, if the two developers in
the inner layer leave, the total infor-
mation loss is more than 60%. This
will cause significant disruptions to
the daily operations of the team.
In comparison, since Hadoop
has a flat team structure as shown
in Figure 1a, we calculate the aver-
age IL dev of the top four develop-
ers, which is only 0.187. This implies
much more affordable information
loss from an individual develop-
er’s absence, compared to the other
projects.
Evaluation
We mined the revision history of
each project to calculate the actual
J A N U A R Y/ F E B R U A R Y 2 0 1 8
No. of developers No. of emails
88  1,479
64 18,675
16 12,105
43  3,971
63 19,613
42 17,281
code contribution percentage made
by each core developer (if any core
developers exist) to the project
code base. If core members contrib-
ute the majority of changes to the
code base, these core developers are
indeed the point of risk for team ro-
bustness. In other words, when those
core developers become unavailable,
daily code revisions will be signifi-
cantly disrupted. In contrast, if con-
tributions by core members are less
significant, the project’s daily opera-
tions would not be affected much.
Table 2 lists the evaluation results.
The second column shows the total
number of code revisions submitted
by the entire development team of
a project during the studied period.
Columns 3 to 6 list the contribution
percentage of each inner-layer mem-
ber in a project. The core members
are in the inner layer of the calcu-
lated TLCH as shown in Figure 1.
The last column shows the total con-
tribution percentage of all the inner-
layer members. Since Hadoop has a
flat structure, the last row in Table 2
just lists the maximal, average, and
standard deviation of the individual
developer’s contributions.
We can make the following ob-
servations from Table 2. First, the
few inner-layer members (up to four
| I E E E S O F T WA R E 67
 FOCUS: ACTIONABLE ANALYTICS

100 100
Average IL, 0.187 Core member IL, 0.464
80 80

Information loss (%)

Information loss (%)

60 60

40 40

20 20

0 0
0 1 2 3 4 5 0 1 2 3 4 5
(a) Top developers (b) Top developers

100 100
Core member IL, 0.346 Core member IL, 0.473
80 80
Information loss (%)
Information loss (%)

60 60

40 40

20 20

0 0
0 1 2 3 4 5 0 1 2 3 4 5
(c) Top developers (d) Top developers

100 100
Core member IL, 0.435 Core member IL, 0.395
80 80
Information loss (%)

Information loss (%)

60 60

40 40

20 20

0 0
0 1 2 3 4 5 0 1 2 3 4 5
(e) Top developers (f) Top developers

FIGURE 2. Team information loss (IL) with absent developers. (a) Hadoop. (b) Cassandra. (c) CXF. (d) PDFBox. (e) Camel. (f) HBase.

developers) together make a signifi- be disrupted significantly—who can Second, in comparison, in Hadoop,
cant contribution—from 49% (Cas- replace them and make the large the individual developer usually makes
sandra) to 61% (PDFBox)—to the percentage of revisions? In particu- a relatively trivial percentage of re-
code base of each project. This im- lar, the top core member alone con- visions. The maximal individual
plies that when these few developers tributes 26% (Cassandra) to 49% contribution is 5%, and the indi-
become unavailable, the daily code (PDFBox) of the revisions to the vidual contribution average is only
revision of the projects will likely entire code base. 0.69%, with a standard deviation

68 I E E E S O F T WA R E | W W W. C O M P U T E R . O R G / S O F T W A R E | @ I E E E S O F T WA R E
of 1%. This implies that Hadoop
indeed has the most robust team
structure.
In summary, the data show that
the TLCH of the developing team
and the information loss associated
with the top four developers cal-
culated by our approach can faith-
fully reflect the actual code base
contributions of the core develop-
ers. Thus, our approach can provide
useful insights for evaluating team
robustness.
Limitations and
Future Work
In this article, we analyzed email data
to construct developers’ collabora-
tion links. Developers’ collaboration
can take other forms—for example,
bug-tracking systems, shared code
ownership, and so on. We acknowl-
edge the limitation of considering
only email exchanges. However, our
proposed approach can be general-
ized to collaboration links extracted
from other data sources.
When calculating the TLCH,
the inner layer is distinguished
from the outer layer if the weight
of a node is one standard deviation
above the mean. “One standard de-
viation” is not the only feasible ap-
proach. Users are suggested to tune
this value depending on the project
circumstances.
We plan to address the following
in our future work:
• Here, we apply our approach to
the six Apache open source proj-
ects. We plan to evaluate and
apply this approach on a broader
spectrum of projects with more
diverse characteristics.
• Currently, the TLCH is con-
structed based on data for a
selected period of time and
thus is static. We plan to apply
Table 2. Core developer contributions.
No of. Core 1
Project revisions (%)
Cassandra 23,169 26
CXF 13,393 30
PDFBox  6,204 49
Camel 29,166 44
HBase 13,539 36
Hadoop 16,196 Max. deviation 5 5%, avg. deviation 5 0.69%, and
standard deviation 5 1%
our approach to visualize and
monitor the dynamics of the
collaboration hierarchy over
time, which can provide in-
sights in analyzing a potential
increase or decrease of team
robustness.
• Although the TLCH algorithm
reveals meaningful team struc-
ture for the six projects with
16 to 88 developers, we ac-
knowledge that it may not work
properly for an ultra-large-scale
development team with more
than hundreds of developers.
But, based on similar rationale,
we plan to extend the TLCH
to a Multilayer Collaboration
Hierarchy to describe the team
structure.
Related Work
In the past decades, numerous stud-
ies examined the social structure
of software development teams us-
ing different methods for different
goals. 5–10 This section compares
this article with the most relevant
prior work.
Christian Bird and his colleagues
mined the social network from
the public email archive of Apache
J A N U A R Y/ F E B R U A R Y 2 0 1 8
Core 2 Core 3 Core 4
(%) (%) (%) Total (%)
12 9 2 49
28 N/A N/A 58
12 N/A N/A 61
14 2 N/A 60
12 6 N/A 54
N/A
httpd. 5 They found that the mes-
sages sent by an individual and the
number of source changes that in-
dividual makes have a Spearman’s
rank correlation of about 0.8. This is
consistent with our evaluation results:
the top few inner-layer developers in
each project contribute a majority
of the code revisions. Chris Jensen
and Walt Scacchi proposed an “on-
ion” diagram to represent the dif-
ferent roles of open source software
developers: active users, develop-
ers, project managers, community
managers, core developers, passive
users, and observers.7 Andrew Meneely
and Laurie Williams found that de-
veloper social network measures,
such as the edges, social distance,
and network centrality, are consis-
tent with developers’ perception of
their actual collaborations.8 Gustavo
Oliva and his colleagues reported
that only 25% of the developers in
a project may be considered as key
developers, who are often active in
the mailing list and fulfill the coor-
dination requirements.9 Recently,
Mitchell Joblin and his colleagues
reported that network metrics are a
better data source to capture the core
developers in a project, compared
| I E E E S O F T WA R E 69
 FOCUS: ACTIONABLE ANALYTICS

managers can leverage this approach

ABOUT THE AUTHORS

to periodically monitor the robust-

LU XIAO is an assistant professor in the Software Engineer- ness of the development team and
ing Division of the School of Systems and Enterprises at identify the points (developers) of
Stevens Institute of Technology. Her research interests include risk for team robustness. A potential
software architecture, software evolution and maintenance, solution to improve robustness could
software economics, and software ecosystems. Xiao received be regular work rotation or improved
a PhD in computer science from Drexel University. Contact her collective code ownership.
at lxiao6@stevens.edu.
References
1. B.W. Boehm, “Software Risk Man-
ZHONGYUAN YU is a research assistant professor in agement: Principles and Practices,”
Stevens Institute of Technology’s School of Systems and IEEE Software, vol. 8, no. 1, 1991,
Enterprises. Her research interests include applied statistics, pp. 32–41.
data visualization, simulation, text mining, and socioeconom- 2. A. Cockburn and J. Highsmith, “Ag-
ics to facilitate strategic decision making. Yu received a PhD ile Software Development, the People
in system engineering from Stevens Institute of Technology. Factor,” Computer, vol. 34, no. 11,
Contact her at zyu7@stevens.edu. 2001, pp. 131–133.
3. E. Otte and R. Rousseau, “Social
Network Analysis: A Powerful Strat-
BOHONG CHEN is a master’s student in business intelligence egy, Also for the Information Sci-
at Stevens Institute of Technology. His research interests are ences,” J. Information Science,
social network analytics, process optimization, and pattern vol. 28, no. 6, 2002, pp. 441–453.
recognition. Chen received a bachelor of engineering from 4. B. Iglewicz and D.C. Hoaglin, How
Shanghai Normal University. Contact him at chenbohong@ to Detect and Handle Outliers, ASQ
hotmail.com. Press, 1993.
5. C. Bird et al., “Mining Email Social
Networks,” Proc. 2006 Int’l Work-
XIAO WANG is a PhD student in Stevens Institute of Tech- shop Mining Software Repositories
nology’s School of Systems and Enterprises. His research (MSR 06), 2006, pp. 137–143; doi.
interests are software architecture and software evolution and acm.org/10.1145/1137983.1138016.
maintenance. He received a master’s in electrical engineer- 6. C. Bird et al., “Latent Social Struc-
ing from Stevens Institute of Technology. Contact him at ture in Open Source Projects,” Proc.
xwang97@stevens.edu. 16th ACM SIGSOFT Int’l Symp.
Foundations of Software Eng.
(SIGSOFT 08/FSE 16), 2008, pp.
24–35; doi.acm.org/10.1145/1453101
.1453107.
7. C. Jensen and W. Scacchi, “Role Mi-

T
to count-based metrics such as he study results suggest that
churn (changed lines of code) and our approach can effectively
commits.10 help people intuitively un-
Compared to the work we just derstand and quantitatively evalu-
mentioned, the uniqueness of this ate the robustness of a development
article is that it leverages the analy- team. Even though our approach
sis of the developer social network has so far been applied only on open
for a new angle: analyzing team source projects, it is directly appli-
robustness. cable to commercial projects. Project
gration and Advancement Processes
in OSSD Projects: A Comparative
Case Study,” Proc. 29th Int’l Conf.
Software Eng. (ICSE 07), 2007, pp.
364–374; dx.doi.org/10.1109/ICSE
.2007.74.
8. A. Meneely and L. Williams, “Socio-
technical Developer Networks:
Should We Trust Our Measurements?,”
Proc. 33rd Int’l Conf. Software Eng.

70 I E E E S O F T WA R E | W W W. C O M P U T E R . O R G / S O F T W A R E | @ I E E E S O F T WA R E
 (ICSE 11), 2011, pp. 281–290; and Technology (CRIWG 12), 39th Int’l Conf. Software Eng. (ICSE
doi.acm.org/10.1145/1985793 2012, pp. 97–112; dx.doi. 17), 2017, pp. 164–174.
.1985832. org/10.1007/978-3-642-33284-5_8.
9. G.A. Oliva et al., “Character- 10. M. Joblin et al., “Classifying Develop- Read your subscriptions
izing Key Developers: A Case ers into Core and Peripheral: An Em- through the myCS
publications portal at
Study with Apache Ant,” Proc. pirical Study on Count and Network
18th Int’l Conf. Collaboration Metrics,” Proc. 2017 IEEE/ACM
http://mycs.computer.org

IEEE COMPUTER GRAPHICS AND APPLICATIONS

IEEE COMPUTER GRAPHICS AND APPLICATIONS
IEEE COMPUTER GRAPHICS AND APPLICATIONS

IEEE COMPUTER GRAPHICS AND APPLICATIONS

November/December 2016
July/August 2016 September/October 2016 January/February 2017

Quality
November/December 2016
September/October 2016

January/February 2017

Assessment
July/August 2016

Defense

C G &A
and
Perception Applications
Quality Assessment and Perception in Computer Graphics

in Computer Graphics
Water, Sky, and the Human Element
Sports Data Visualization

Defense Applications

VOLUME 37 NUMBER 1
VOLUME 36 NUMBER 5
VOLUME 36 NUMBER 4

VOLUME 36 NUMBER 6

c1.indd 1 12/14/16 12:21 PM

c1.indd 1 6/22/16 1:20 PM c1.indd 1 8/22/16 2:59 PM

c1.indd 1 10/24/16 3:44 PM

www.computer.org/cga
IEEE Computer Graphics and Applications bridges the theory
and practice of computer graphics. Subscribe to CG&A and
• stay current on the latest tools and applications and gain
invaluable practical and research knowledge,
• discover cutting-edge applications and learn more about
the latest techniques, and
• benefit from CG&A’s active and connected editorial board.

J A N U A R Y/ F E B R U A R Y 2 0 1 8 | I E E E S O F T WA R E 71