Towards Recognizing and Rewarding Efficient

Developer Work Patterns

Will Snipes, Vinay Augustine
ABB Corporate Research
Industrial Software Systems
Raleigh, NC USA
Email: will.snipes/vinay.augustine@us.abb.com
Anil R. Nair
ABB Corporate Research
Industrial Software Systems
Bangalore, India
Email: anil.nair@in.abb.com
Emerson Murphy-Hill
North Carolina State University
Department of Computer Science
Raleigh, N.C. USA
Email: emerson@csc.ncsu.edu

Abstract—Software engineering researchers develop great tech-
niques consisting of practices and tools that improve efficiency
and quality of software development. Prior work evaluates
developers’ use of techniques such as Test-Driven-Development
and refactoring by measuring actions in the development environ-
ment. What we still lack is a method to communicate effectively
and motivate developers to adopt best practices and tools.
This work proposes a game-like system to motivate adoption
while continuously measuring developers’ use of more efficient
development techniques.
I. I NTRODUCTION
Software engineering researchers define techniques consist-
ing of practices and tools that improve efficiency and quality
of software development. Techniques such as refactoring with
the aid of tools like Resharper, unit testing with NUnit or
CppUnit tools may be well known but not necessarily widely
or frequently used.
Deploying these techniques requires communicating and
motivating developers to use them. Murphy-Hill and col-
leagues [1] show that developers can effectively learn about
new tools from their peers by watching them work and through
recommendations, yet this happens very rarely, in part because
developers rarely work on the same computer at the same
time. This personal connection limits the knowledge transfer
to the co-located circle of colleagues connected with the early
adopters.
Finding a way to automate the communication of technique
recommendations between developers enhances deployment of
best practices in a globally distributed software development
organization. To best match the peer recommendation scenario
described above, the method should mimic the personal rec-
ommendation model while rewarding successful adopters.
In this paper we envision an automated technique recom-
mendation system composed of three aspects. First, the system
will monitor work patterns of developers working on tasks.
Second, the system will heuristically compare work patterns
of developers with those using a priori defined practices
and tools that are more efficient for the pattern. Third, the
system will make recommendations in the context of a game
that motivates developers to adopt new techniques through
personal and situation-based recommendations. A system that
recognizes developers who adopt suggested techniques could
give a significant boost to deploying best practices.
II. U NDERSTANDING W ORK PATTERNS
Developers work on tasks such as implementing new fea-
tures, making enhancements or fixing bugs. Tasks can be
divided into development sessions [2] where developers may
extend a method, refactor some code, navigate around the
code, run the debugger, and make modifications to the program
structure.
Because we are interested in how the developer does
the work in each type of session, we require a more fine
grained detail that describes the steps taken by a devel-
oper. For example a sequence within a navigation session
where developers are finding relevant code artifacts for a task
is Start → OpenDeclaration → V iewRef erences →
ShowInstruction → End. We combine multiple sessions
of the same type into a set of sessions that form a pattern.
A work pattern is thus defined as the set of sequences the
developer uses for a particular session type. Using the work
pattern sequences we can evaluate how the developer navigates
the code, the steps they use to debug, or at a higher level,
whether they are following a development best practice such
as Test-Driven-Development. By comparing the work patterns
across multiple sessions of the same type, we identify efficient
and inefficient work patterns and may recommend ways to
increase their efficiency.
III. O BSERVING W ORK PATTERNS
Collecting fine-grained data by monitoring how developers
work requires a system such as Mylyn Monitor [3], HackyStat
[4], or PROM [5]. These systems collect command and click-
stream data from the developer’s workstation in a log format.
The level of detail required for identifying work patterns
includes the commands issued in the development environ-
ment, the commands issued editing the code, and commands
issued outside the environment (e.g. testing or discovering
information). Mylyn Monitor captures the commands issued
in the Eclipse IDE, HackyStat captures the command events
in multiple tools through plug-ins, and PROM collects com-
mands performed in the IDE using plug-ins. Thus prior work
demonstrates the capabilities for collecting fine-grained data
on developer work activities.
 IV. A NALYZING W ORK PATTERNS
Our hypothesis is that software developers’ work patterns
reveal when developers are using effective techniques for a
task. Furthermore, we hypothesize that by suggesting training
or alternative ways of working, we can move developers to
more effective techniques.
Space prevents us from an exhaustive comparison to prior
work, but three areas are especially relevant to analyzing
pattern data from developer environments. One approach is to
create a Markov chain of the steps taken during a development
session [6]. Another is to leverage recent research in the web
community in clickstream analysis. Much of the research in
this area is devoted to identifying similarities [7], user profiles
[8], and usage patterns [9].
Describing work patterns in finite-state Markov chains using
the relevant statistical properties is useful for comparison
and reveals key attributes of efficient and inefficient work
patterns. We demonstrate this analysis by following the prac-
tices described in [6] using JUMBL [10] for describing the
usage model and for calculating the occupancy and frequency
statistics of the model.
In our preliminary data analysis we identified work patterns
from data collected by monitoring developers performing a
maintenance task [11]. The study provides a data set generated
by assigning a task to fix the same bug to multiple developers.
A video of the activity records the developer actions that
are translated to a text log file of steps taken while editing
the code. The data from this study measures how long each
developer took to perform the maintenance tasks as well as
how well they understood the code they were maintaining.
Because they are working on the same task, we can compare
the steps each developer takes to solve the bug.
To build the Markov model, we convert the step log file to a
graphical representation where each unique step is represented
as a unique node and each adjacent pair of nodes are repre-
sented as a connection between the corresponding nodes. i.e., a
connection from node A to node B exists if the developer uses
the command B following A in the log file. Figure 1 shows
the model for developer Searcher. The edges are labeled with
the name of the connected nodes and the number of times this
transition has occurred. For example, Open Type occurs a two
times after Scan in the log file, and the edge from node Open
Type to Scan in the graph is labeled with Open Type {2} .
We highlight the differences between patterns by selecting
two developers from a set of 12 where the developer we call
“Searcher”used structured navigation to find the bug and the
developer we call “Browser” used unstructured and browsing
more. Structured navigation defined in [12] is using code
search commands such as Find References to locate important
code whereas unstructured navigation the developer browses
the code or uses Find Text commands for the same. For the
navigation patterns, we investigate some key metrics motivated
by the data. The number of nodes in the model and the number
of arcs describe the breadth of commands used to perform
the task. The average sequence length and sequence length
TABLE I
S EQUENCE DATA A NALYSIS
Developer Searcher Browser
Navigation nodes 17 27
Navigation arcs 37 89
Average sequence length 6.3 15.5
Sequence length standard deviation 2.5 8.7
TABLE II
NAVIGATION S TRUCTUREDNESS
Developer Searcher Browser
Neither 44% 19%
Structured 38% 18%
Unstructured 18% 63%
variance describe how many steps the developer takes for the
average session. We also look for the most frequently issued
commands within a session to identify repetitive steps, and
look at characteristics of the commands to determine whether
the developer is using more or less effective commands.
In Table I we show data supporting how using structured
navigation is more effective to isolate the bug in fewer steps
than unstructured. For Searcher, the average number of navi-
gation session steps (average sequence length per navigation
session) is 6.3 and the standard deviation in the number of
steps is 2.5. Browser used more than twice the average number
of steps per session used by Searcher. The higher standard
deviation indicates that Browser was also less consistent in
the number of steps taken for each navigation session. While
not conclusive, Browser’s methods to do the same task seem
less efficient than Searcher’s methods though this could be
influenced by differences in their experience level or other
factors not measured.
Investigating the commands issued during navigation activi-
ties highlights opportunities to improve. In the model solution,
we classified each navigation command as structured or un-
structured and counted the number of times each command
was issued along with time spent in each step. Table II shows
the difference in the distributions between the developer’s
use of structured vs. unstructured navigation. Searcher used
structured navigation twice as often as unstructured, where
Browser used Unstructured three times more than structured.
By analyzing the repetitive actions that occur in a sequence
we determine whether a developer is using ineffective com-
mands. The stimulus describes the trigger of a transition to a
state, in this case, issuing a specific command such as Find
Text. Find Text locates a string in a source file and is less
effective than Find References that searches the entire project
or solution. In our two developers, we see Browser using find
text on average four times per navigation session. In contrast,
Searcher does not use any command more than once on
average per navigation session. These key indicators highlight
opportunities to help Browser improve and become a more
effective developer. Recommending a structured navigation
approach to program investigation to Browser could make
them more successful according to Robilliard et al. [12].
 Fig. 1. Work Pattern Markov Model for Developer Searcher
V. M OTIVATING I MPROVEMENTS TO W ORK PATTERNS
Applying the information collected and analyzed about de-
veloper work patterns must help developers improve. Helping
to improve implies that we provide information on how to
improve and motivate the developer to adopt the new patterns
in a positive way.
Providing a means for recognizing achievements of the
developer towards improvement is one strategy to generate the
proper motivation. This approach is inspired by the gamifica-
tion trend where processes are enhanced by game mechanics
to motivate people to follow them. A study by Saatchi and
Saatchi S reporting on the gamification trend [13] claims that
75% of employed 18-45 year old respondents would be at least
somewhat interested in working in an environment that uses
games to increase productivity.
Since the Saatchi and Saatchi S study did not focus on
software developers, we wonder whether achievement could
also motivate people in knowledge work environments such as
software development. Maehr states in a study of achievement
motivation in foreign cultures that since these behaviors can be
triggered by circumstance, they are normally present in most
people [14]. He further states that “achievement motivation is
universal...in all people and that it only needs establishing of
the right cues and context to elicit these behaviors”. Therefore
it is possible that developers from different backgrounds
can be motivated by a form of game that recognizes their
achievements.
To verify, we plan to conduct a study of whether this
approach is acceptable to developers. The study will determine
the following:
1) Whether developers are open to the idea of applying
achievement motivation to their daily work activities.
2) Whether developers accept the intrusion of automation
to monitor their work patterns in order to receive the
feedback and achievement recognition.
3) Whether recommendations of their peers would encour-
age them to adopt new tools and practices
Then we plan to create a recognition system within the de-
velopment environment. The system will recognize developers
who follow good practices with achievement rewards such as
badges. The system could provide suggestions to developers
who are striving to improve by suggesting that using specific
techniques, practices, or tools would improve their score.
Recognizing developers who try new techniques may provide
the extra motivation necessary to move developers up the
efficiency scale.
Figure 2 shows a mock-up of an example achievement
recognition system. The developer earned three badges of
different types. The first is for following Test-Driven Develop-
ment practice, second is for using Sando [15] for code search,
and third is for using breakpoints in the debugger. As new tools
and practices are supported by the analysis, new badges can be
provided by the system to developers who use them. With this
system we plan to measure changes in developer practices and
tool use in order to determine whether the system is having a
positive effect.
VI. R ELATED W ORK
Researchers have developed systems that aim to answer
specific questions using data from command logs. Zorro
[16] leverages the HackyStat data to determine if developers
are adhering to test-driven development. The tool employs
a similar approach dividing the work of the developer into
episodes delineated by when test cases pass. The researchers
define multiple types of episodes then determine whether
the developer uses Test-Driven-Development or not in each
episode.

Fig. 2. UI Mockup with Achievement Badges
Murphy-Hill, Parnin, and Black [17] use the Mylyn Monitor
to explore whether or not developers use the automated
refactoring tools present in Eclipse. They look for specific
refactoring commands in Eclipse and determine the amount
of time developers use tools versus hand refactoring the code.
Robillard, Coelho and Murphy explore hypotheses around
how developers can be more effective at performing a main-
tenance task [12]. Key conclusions are that developers are
more more successful finding and fixing bugs when they create
a detailed plan for implementing a change, use structured
navigation with keyword search or cross-reference search, and
only review methods once during their search.
Building on this success, this work proposes a general-
ized method to automate the identification of the techniques
described in the related work and others, and determine
when the techniques would help a developer improve their
work patterns. Then we intend to create a game-like virtual
environment where developers learn from each other and are
motivated to adopt the best practices developed by these and
other researchers.
VII. C ONCLUSION
We propose an automated system for capturing developer
work patterns, suggesting improvements, and motivating adop-
tion of those improvements. This system will leverage existing
tools in order to monitor developer work patterns. It will apply
data mining and statistical techniques in order to compare
and model those work patterns and suggest improvements. For
example, the time and number of steps required to navigate
between edits can be compared between developers. Finally,
it will drive adoption of new practices and tools by using an
achievement-based system.
There are a number of key research questions that we hope
to answer. First, can we automatically discover and group work
patterns from recorded developer actions? Does the automated
system to capture work patterns provide relevant data for
analysis? Can analysis of work patterns from multiple develop-
ers identify practices that increase developer efficiency? Are
graph mining techniques effective at identifying similar and
different work patterns such that automated recommendations
are possible? Will developers accept recommended relevant
techniques from an automated system in place of their existing
work patterns? Finally, will a game-based achievement system
drive adoption of new techniques?
This system can be used to study the efficacy of rec-
ommended work patterns. Researchers can use the proposed
system to measure whether or not their tools lead to increased
efficiency for specific tasks by comparing the work patterns
of developers before and after a new pattern is adopted.
ACKNOWLEDGMENT
Thanks to David Shepherd, Thomas Fritz, and Christoph
Bäunlich for providing developer activity logs from their study
[11]. JUMBL is developed and maintained by the University
of Tennessee Software Quality Research Laboratory.
R EFERENCES
[1] E. Murphy-Hill and G. C. Murphy, “Peer interaction effectively, yet in-
frequently, enables programmers to discover new tools,” in Proceedings
of the ACM 2011 Conference on Computer supported cooperative work,
ser. CSCW ’11. ACM, 2011, pp. 405–414.
[2] R. Robbes and M. Lanza, “Characterizing and Understanding Devel-
opment Sessions,” in 15th IEEE International Conference on Program
Comprehension, ser. ICPC ’07. IEEE, Jun. 2007, pp. 155–166.
[3] G. C. Murphy, M. Kersten, and L. Findlater, “How are Java software
developers using the Elipse IDE?” IEEE Software, vol. 23, no. 4, pp.
76–83, Jul. 2006.
[4] P. M. Johnson, H. Kou, J. Agustin, C. Chan, C. Moore, J. Miglani,
S. Zhen, and W. E. J. Doane, “Beyond the personal software process:
Metrics collection and analysis for the differently disciplined,” in Pro-
ceedings of the 25th international Conference on Software Engineering.
IEEE Computer Society, 2003.
[5] A. Sillitti, A. Janes, G. Succi, and T. Vernazza, “Collecting, integrating
and analyzing software metrics and personal software process data,” in
Proceedings of the 29th Euromicro Conference. IEEE, 2003, pp. 336–
342.
[6] S. Prowell, Cleanroom software engineering: technology and process.
Reading, Mass: Addison-Wesley Professional, 1999.
[7] A. Banerjee and J. Ghosh, “Clickstream Clustering Using Weighted
Longest Common Subsequences,” in Proceedings of the Web Mining
Workshop at the 1st SIAM Conference on Data Mining, 2001.
[8] O. Nasraoui, M. Soliman, E. Saka, A. Badia, and R. Germain, “A
Web Usage Mining Framework for Mining Evolving User Profiles
in Dynamic Web Sites,” IEEE Transactions on Knowledge and Data
Engineering, vol. 20, pp. 202–215, Feb. 2008.
[9] R. Atterer and A. Schmidt, “Tracking the interaction of users with
AJAX applications for usability testing,” in Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, ser. CHI ’07.
ACM, 2007, pp. 1347–1350.
[10] S. J. Prowell, “JUMBL: a tool for model-based statistical testing,” in
Proceedings of the 36th Annual Hawaii International Conference on
System Sciences, 2003., vol. 9. IEEE, Jan. 2003, pp. 6–9.
[11] D. Shepherd, T. Fritz, and C. Bäunlich, “Supporting search and naviga-
tion through code context,” Zurich, Switzerland, Tech. Rep., 2012.
[12] M. P. Robillard, W. Coelho, and G. C. Murphy, “How effective devel-
opers investigate source code: an exploratory study,” IEEE Trans. Softw.
Eng., vol. 30, pp. 889–903, Dec. 2004.
[13] Saatchi and Saatchi S. (2011, Jun.) Gameification
for business brands and loyalty. [Online]. Available:
http://www.slideshare.net/Saatchi S/gamification-study
[14] M. Maehr, “Culture and achievement motivation.” American Psycholo-
gist, vol. 1974, no. 29, pp. 887–896, 1974.
[15] D. Shepherd, K. Damevski, B. Ropski, and T. Fritz, “Sando: An
extensible local code search framework,” in Proceedings of Foundations
of Software Engineering, Tool Demo. ACM, 2012.
[16] P. M. Johnson and H. Kou, “Automated Recognition of Test-Driven
Development with Zorro,” in Agile Conference (AGILE), 2007. IEEE,
Aug. 2007, pp. 15–25.
[17] E. Murphy-Hill, C. Parnin, and A. P. Black, “How We Refactor, and How
We Know It,” IEEE Transactions on Software Engineering, vol. 38, pp.
5–18, Jan. 2012.