Implementing an Internet-Enabled Software Experience Factory:

Work in Progress

Richard Webby
Centre for Advanced Empirical Software Research, School of Information Systems, University of New
South Wales, Sydney 2052, Australia. Email: r.webby@unsw.edu.au

Carolyn Seaman
University of Maryland, Baltimore County, Department of Information Systems, Baltimore, MD
21250, USA. Email: cseaman@umbc.edu

Manoel Mendonca
Department of Computer Science, University of Maryland, College Park, MD 20742, USA. Email:
manoel@cs.umd.edu

Victor Basili
Fraunhofer Center for Experimental Software Engineering, Maryland, and Department of Computer
Science, University of Maryland, College Park, MD 20742, USA. Email: basili@cs.umd.edu

Yong-Mi Kim
Q-Labs, Inc., 4511 Knox Road, Suite 305, College Park, MD 20740, USA. Email: Yong-Mi.Kim@q-
labs.com

[This research was in part supported by Maryland Industrial Partnerships grant no. 2015.22]

Introduction
Many companies already have informal or semi-formal methods of recording and sharing experiences
about software development within their organizations. They may hold meetings and conferences on a
regular basis to discuss and share experiences informally. They may record project data in project
binders or in computerized databases. They may even write “lessons learned” documents to record
subjective experiences with projects. However, typically much of the information about experiences is
not systematically stored or packaged in a form for effective retrieval and reuse, and consequently is
never used. Moreover, the realisation of personnel that much of the data will never be accessed creates
a negative feedback loop where the valuable experiences are not recorded in the first place.

The experience factory approach aims to establish an organizational infrastructure to facilitate

systematic and continuous organizational learning through the sharing and reuse of experiences in
software engineering [Basili et al. 1994]. It involves setting up a group separate to the development
teams, called the experience factory, which is responsible for collecting experiences from development
projects, packaging the experiences by building empirical models and structuring informal knowledge,
and validating and spreading experience packages into development projects. It aims to install
processes and techniques into organizations to encourage data collection, facilitate packaging of
experiences, and promote the dissemination (push) and retrieval (pull) of experiences.

Although a number of software engineering organizations have adapted some of the principles of
experience factories into their own businesses, very few have incorporated the entire concept and
published their experiences with the approach. Two that have are the Software Engineering Laboratory
at NASA [Basili et al, 1995] and Daimler Benz [Houdek et al,1998]. In both cases, the establishment
of experience factories required substantial effort and commitment on the part of management and
development staff. For an experience factory to work successfully over the long term, an organization
must instil a new philosophy of organizational learning, establish an organizational structure and
processes for the experience factory to collect, package and share experiences, and baseline the
organization to identify process improvement needs. Once in place, the experience factory will also
require substantial ongoing effort and commitment to maintain as an effective agent for continuous
software process improvement. Perhaps the main reason for the slow adoption of the experience factory
approach is its cost. We believe that the emerging technologies of the Internet offer great potential to
support the establishment and maintenance of experience factories in organizations. However, as with
any technology, we must first understand how it can be best applied in industrial settings.

Consequently this paper reports preliminary results and planned experiments from a research project
aimed at implementing an Internet-enabled experience factory within an industrial setting. Q-Labs, a
globally distributed software improvement consultancy, wished to implement an organization-wide
experience factory to share and distribute experiences about consulting projects among their staff. Q-
labs recognized that their information sharing needs went beyond basic Internet communication (e.g. e-
mail and ftp) to the need to share knowledge at a higher level (i.e. “packaged” information). They
needed to develop the infrastructure (both organizational and technological) to support the collection,
distribution and retrieval of such knowledge in a cost effective manner. The experience factory
approach was chosen as the basis for the organizational infrastructure, and Internet technologies such
as Java and secure web servers, were selected as the basis for the technological infrastructure.

This paper will focus on describing technical aspects and experimental plans with respect to the
retrieval interface for the prototype experience factory system under development. The issues of how
to collect, store and package experiences for reuse (i.e. the organizational aspects and details of the data
collection and packaging interfaces) will be left to future papers.

Implementation of the Experience Factory

Before deciding to implement a new tool, we sought to examine existing technology from related
domains. Knowledge management [Liebowitz and Beckman, 1998] is a rapidly growing discipline in
information systems concerned with the use of information technology to collect, organize, store and
share knowledge in organizations. A well-known example of knowledge management technology is
Lotus Notes, which has been successfully applied in many organizational contexts. We examined the
potential of applying these technologies to the specific problem in Q-Labs, but decided they had
inadequate support for retrieval. (Typically the existing knowledge management tools contain only a
basic web-browser style search engine.)

Consequently we looked for an information retrieval approach that would support the complex, multi-
dimensional data associated with software engineering experiences. We found an innovative tool called
Spotfire [Spotfire Inc., 1999], which implements the dynamic querying approach [Schneiderman, 1994]
to rapid data analysis and information retrieval. Spotfire has traditionally been applied to problems of
visual data mining in pharmaceutical, chemical, and biotechnical industries. We examined this
technology and decided it would be worthwhile to investigate its potential as a retrieval interface for an
experience factory. Figure 1 shows an example of the graphical retrieval interface for the Experience
Factory based on Spotfire. The illustration shows the category-based retrieval of Q-Labs’ experience
packages.
Figure 1: Screen shot of the EF Spotfire-based user interface

In this retrieval interface, each data point represents an experience package, which is analogous to a
document in this case. The user is viewing three charts in Figure 1 to provide three different views on
the same set of documents. Packages are classified in categories based on sets of relevant attributes. As
shown in the top right-hand side of Figure 1, query devices such as check boxes, radio buttons, and
range bars are used to interactively select the desired packages based on their attribute values. Packages
can be retrieved for inspection, once a reasonable subset of packages is (de)selected using the query
devices. As shown in the bottom right-hand side of Figure 1, the retrieval interface provides the ability
to click on a data point and jump to a generated web page containing the complete description of the
experience package, as well as links to its files and other related experience packages.

The original version of Spotfire unfortunately had a number of limitations:

1. It was not platform independent; it only ran under Microsoft Windows.

2. It was difficult to create and integrate different types of packages under one application because
different sets of attributes are needed to classify different types of packages. Figure 1, for example,
shows the packaging of documents for search and retrieval. The attributes in this interface do not,
however, support the packaging of projects. It also does not support establishing the necessary
links between documents and projects.
3. It also did not provide adequate functionality for search and retrieval of distributed packages, one
of the key requirements of the Experience Factory. For example, it was not possible in the first
version to click on a data point and start a web browser with a given URL.

In order to solve these limitations, we are reimplementing the system in Java to ensure cross-platform
capability, support for different types of packages, and ease of access across an Intranet or the Internet.
As shown in Figure 2, the architecture of the system follows a three-tier model.
 EF DQI EF DQI EF DQI ... ... ...
UI for a chosen
Remote Remote package class
machine machine

EF-proprietary
IP Network protocol

EF Server Server
Standard machine
protocol
JDBC

DBMS-proprietary
protocol
Database
Comercial server
DBMS EF Repository

Figure 2: Experience Factory Architecture

In this architecture, the dynamic query user intefaces (EF DQIs) provide integrated support to multiple
package classes. They work as client applications sending requests to a "middle tier" of services. This
“EF Server” receives messages from the EF DQIs and translates them into low-level SQL (Standard
Query Language) calls to a “EF Repository.” The EF Repository stores all the information necessary
for the EF operation in a relational database managed by a commercial DBMS (Data Base Management
System.) Its current implementation uses Oracle 8 under Windows NT but, due to the use of the
standard JDBC (Java database connectivity), it can easily be ported to any other major DBMS.

The three-tier model makes it possible to maintain complete control over access and the kinds of
updates that can be made to the EF data. The EF Repository provides a common representation model
for all EF packages, independent of their type. All packages are represented by sets of files and
attributes in the relational database tables. In the future, we plan to improve upon the current repository
implementation of flat-table structures that only use attributes to characterize experience packages. The
ultimate plan to move to more sophisticated structures which might be based on XML (Extendible
Markup Language). XML provides the potential to define customized mark-up languages to encourage
knowledge sharing within or among organizations.

Experimental Plans

This approach is being implemented with Q-Labs, a multinational software improvement consultancy
organization with offices in Sweden, Norway, Germany, and the USA. We are planning a program of
experimental trials in Q-Labs, beginning with a pilot study. The pilot study has the following goals:

1. To evaluate the current set of attributes in terms of completeness, usefulness, and clarity.
a. Are there any attributes that would be useful to include?
b. Are there any attributes that are not clear in their meaning?
c. What attributes are most used in searches?

2. To evaluate the Spotfire-based retrieval interface in terms of usefulness, usability, and

appropriateness.
a. Are subjects able to find what they are looking for using the interface?
b. Do subjects get frustrated by the interface?
c. Are there any suggested improvements?
 3. To evaluate the data entry interface (a simple forms-based interface) in terms of feasibility,
usability, and impact on working procedures.
a. Are all information requests clear and understandable to the user?
b. Does the interface ask for information that is difficult to provide?
c. Does using the interface represent a significant change in the way things are done?

The pilot study will involve three or four Q-Labs personnel in different offices in North America and
Europe, who will be given 10-20 typical retrieval tasks to perform using the tool, as well as a data entry
task drawn from their current project work. Protocol observation will be employed to study the
reactions of the subjects to the tool. The subjects will also be surveyed using structured post-
experiment interviews to gauge answers to the following types of questions:

1. What did you like most about this interface?

2. Was there anything really annoying about using it?
3. Is there any information that would be useful to include in the interface that isn’t there?
4. Are there any attributes that are not clear in their meaning?
5. What attributes did you use most in searches?
6. Did you feel that you were able to find what you were looking for using the interface?
7. How could the interface be improved?
8. Do you think you would use this tool, once the database was populated, in your everyday work?
9. What did you like most about the data entry interface?
10. How was using this interface different from the usual procedure for recording this type of
information? Do you think, in general, that this would save you time or not?

It is hoped to present the results of the pilot study at the workshop.

References
Basili, V.R., M.V. Zelkowitz, F. McGarry, J. Page, S. Waligora, and R. Pajerski (1995). “SEL's
Software Process Improvement Program,” IEEE Software , vol. 12, no. 6, pp 83-87.

Basili, V.R., G. Caldiera and H.D. Rombach (1994). “The Experience Factory,” in J.J. Marciniak (ed),
Encylopedia of Software Engineering, vol. 1, pp 469-476. John Wiley & Sons.

Houdek, F., K. Schneider, and E. Wieser (April 1998). “Establishing Experience Factories at Daimler-
Benz: An Experience Report,” in Proc. of 20th International Conference on Software Engineering,
Kyoto, Japan, pp. 443-447.

Liebowitz, J. and T. Beckman, (1998). Knowledge Organizations: What Every Manager Should Know.
St. Lucie Press.

Shneiderman, B. (1994). “Dynamic Queries for Visual Information Seeking,” IEEE Software, vol. 6,
no. 11, pp 70-77.

Spotfire, Inc (1999). “Spotfire Pro 3 – Third Generation in Interactive Data Visualization,” World
Wide Web, http://www.spotfire.com/products/spotfire/spotfire3/