Overview

Process
Criteria
Resources

Evaluating, Selecting, and Implementing

Free/Libre/Open-Source Software

Marko Schütz

Department of Mathematical Sciences

University of Puerto Rico at Mayagüez
Mayagüez, PR

07 May 2009

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Outline

1 Overview

2 Process

3 Criteria

4 Resources

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Workshop Structure

– presentation of background
– independent work
– review

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Introduction
– the problem
> FLOSS sought to fill some need
> wealth of FLOSS available: 100,000s of projects
– stages of the process
requirements

implement identify candidates

analyze review

compare

– criteria
– resources supporting the process
> existing evaluation frameworks
> project documentation templates
Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
 Overview
Process
Criteria
Resources

Stages

involve user- and developer community

– clarify and document requirements
– identify candidates
– review candidates
– shortlist and prepare comparison
– analyze results
– implement

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Clarify and Document Requirements

– elicit
> user interviews
> user surveys
> brainstorming sessions

– FLOSS development differs from proprietary development

> FLOSS: do one thing right and inter-operate
> proprietary: cover many features, use lock-in (data formats)
to make users stay within application
> formulate neutral requirements

By e.g. requiring a single application for all identified needs

many possible FLOSS combinations will be ruled out
– document use cases

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Identify Candidates

– perform search on the Internet, including specialized sites:

http://sourceforge.net, http://directory.fsf.org,
http://freshmeat.net, http://debian.org,
http://icewalkers.com, http://cpan.org.
– research individual features/requirements to find
candidates
– use name of well-known proprietary software with the term
“open source” on google

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Identify Candidates (cont.)

– use existing lists

> A MITRE 2003 report lists 115 FLOSS “generally
recognized as safe”
> The ”Generally Recognized As Mature” (GRAM) contains
39 FLOSS
> Les Trophées du Libre
> Qualification and Selection of Open Source Software
> sourceforge.net project of the month, community choice
awards
> problem with all these lists: outdated
- can only be used as indicators

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Review Candidates

– use existing reviews and comparisons

– mailing lists
– user and developer documentation
– project statistics
– code review
– collect findings

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Shortlist and Prepare Comparison

– weed out using must-have, must-not-have requirements,

e.g. must have unencrypted, standards-based backup
format
– feed back findings to user community
– feed back findings to developer community
– goal: less than a handful solutions (might consist of several
programs)

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Analyze Results

– document lessons learned in previous stages

– weigh and evaluate criteria
> absolute
> relative ranking
– obtain approval from intended users, administration,
management, “customer”

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Implement

– invite some early adopters

– prototype alongside current solution
– collect and maintain automated black-box tests for most/all
use cases
– allow for ample transition time
– encourage constant feedback from users/stake-holders

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Criteria

– often criteria are competing

> example: code size vs number of features
– kinds of criteria
> requirements
> project community
> project source code
> project evolution
> deliverables
> trials

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Community

– developers (number of, background of)

– conferences
– project activity (recent/sustained)
some projects have activity bursts and times where they
fall asleep
– size, organization
– leadership
– mentoring
– GSoC, etc

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Source Code
– technologies
> portability hidden issues: Java may be less portable than
e.g. C
> teams existing competence in those technologies
> teams desire, capability to learn these technologies
> extends to e.g.
- programming language
- database system
- desktop environment
- communications protocols
– How well is the implementation documented?
– code size
> is an expenditure!!! bigger is worse, not better
> all that code has to be maintained
Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
 Overview
Process
Criteria
Resources

Project Evolution
– How long does the project exist?
– history
> How has it proceeded since then?
> Have there been prolonged phases of inactivity?
> How has developer community evolved?
> Major obstacles/challenges faced?

– maturity
> Is it based on a tried concept, architecture?
> What are the revisions it went through?
> Will be different for different kinds of applications
> Installations
- What are the showcase deployments/installations?
- How many installations are there?

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Deliverables

– usability
– quality
– security
– feature list: breadth/focus
– documentation

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Deliverables: Usability
– often misused:
> ease of installation, initial startup: valuable for casual, new
users
> frequent/”power” users need growth path, e.g.:
- scripting
- external control
- plug-ins

– setup prerequisites
> which are shared with other software that is likely to be
used? e.g. requires KDE, requires PostgreSQL.
Amortization.
> done once, updates should be rare. Amortize over user
number, usage time.
> estimated time to update to future versions

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Deliverables: Quality
– minor releases & point/patch releases
> criteria for version number increase
> developer oriented might rely of code repository
– open bugs
> bugs need to be reported
> how many reporters?
> what kind of bugs are reported?
> assignment of appropriate priority
> bug-fixing sprints held
> avoidance of regressions
> reactivity to security relevant bugs
> security relevant bugs relative to software class
> what is considered critical?
> as with most criteria here: absolute numbers mean little
Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source
 Overview
Process
Criteria
Resources

Deliverables: Security

– security vulnerabilities
> how many
> how critical
> over what period

– out-of-box configuration secure?

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Deliverables: Feature List

– the project might be too broad then

> risk the team doea not develop sufficient expertise in one
the individual features
> one of the most successful and longlived approaches has
been the UNIX approach of small tools that do one thing,
one thing only and that do that one thing right.

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Trials

– form an early adoption team: a group of users who start

using the software in real productive work: must be users
who want to use the new software
– early adoption team needs to feed back as much as
possible on their experiences, issues, problems, questions,
surprises, etc
– very carfully listen to early adoption team

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Business Readiness Rating (BRR)

– http://www.openbrr.org
– evaluation methodology
– provides
> template
> some sample evaluations

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Qualification and Selection of Open Source Software

(QSOS)

– http://www.qsos.org
– evaluation methodology
– provides
> template
> browser-based software to work with templates
> sample evaluations

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

ReadySET

– http://readyset.tigris.org
– project documentation library
– provides
> XHTML-based forms for software-engineering projects

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Discussion

– BRR and QSOS are often too rigid, e.g.:

> QSOS only allows three values per criterion: no, partial, full
> BRR and QSOS strive for evaluation of criteria independent
deployment context
- e.g. QSOS’s books available
- cannot be compensated with assignment of weights

> both can be applied with regard to deployment context

worst case: loss of comparability
– ReadySET sometimes too software-development oriented
strip out parts too software-development oriented

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source

 Overview
Process
Criteria
Resources

Independent Work

– pick domain you know something about e.g.

> database systems
> photo management software
> course management software
> statistics software
> data visualization
> enterprise resource planning
– develop a realistic scenario
– use the process described

Marko Schütz Evaluating, Selecting, and Implementing Free/Libre/Open-Source