Case Study in Sofware Danilo Martínez Espinoza

Engineering mdmartinez@espe.edu.ec
 Unit 2.2: Collecting Data

• Data Sources
• Methods of Data Collection
• Methods of Data Analysis
 Data Sources

There are several different sources of information that can be used in a case
study. It is important to use several data sources in a case study in order to limit
the effects of one interpretation of one single data source. If the same
conclusion can be drawn from several sources of information, i.e. triangulation
this conclusion is stronger than a conclusion based on a single source.
In a case study, it is also important to take into account viewpoints of different
roles, and to investigate differences, for example, between different projects and
products.
Commonly, conclusions are drawn by analyzing differences between data
sources.
 Data Sources

According to Lethbridge et al. (2005) data collection techniques can be

divided into three levels:
• First degree: Direct methods means that the researcher is in direct
contact with the subjects and collect data in real time. This is the case
with, for example, interviews, focus groups, Delphi surveys (Dalkey and
Helmer 1963), and observations with “think aloud protocols”.
• These methods are mostly more expensive to apply since they require
significant effort both from the researcher and the subjects.
 Data Sources

• Second degree: Indirect methods where the researcher directly collects raw
data without actually interacting with the subjects during the data
collection. This approach is, for example, taken in Software Project
Telemetry (Johnson et al. 2005) where the usage of software engineering
tools is automatically monitored, and observed through video recording.
• An advantage of first and second degree methods is that the researcher can
to a large extent exactly control what data is collected, how it is collected, in
what form the data is collected, which the context is etc.
 Data Sources

• Third degree: Independent analysis of work artifacts where already available

and sometimes compiled data is used. This is, for example, the case when
documents such as requirements specifications and failure reports from an
organization are analyzed or when data from organizational databases such
as time accounting is analyzed.
• These methods are mostly less expensive, but they do not offer the same
control to the researcher; hence the quality of the data is not under control
either, neither regarding the original data quality nor its use for the case
study purpose
• It should also be noticed that the data has been collected and recorded for
another purpose than that of the research study.
 Methods of Data Collection
It is important to carefully decide which data to collect and how to collect the data.
In the beginning of a study it is often hard to decide exactly what data to collect, because
there are so many activities in the research taking place at the same time.
At the time when the objectives of the study are being decided and the detailed research
questions of the study are being defined it is impossible to know everything about the
studied case.
But as the case study progresses and the researcher learns more about the case, the
research develops a greater appreciation of what data is desirable, what data is available,
and what data is feasible to collect and subsequently analyze.
The researchers may need to collect data before they know whether and how that data will
be useful. Also, the researchers may not know, at the start of the study, about all of the data
that could be collected and, therefore, the researchers are likely to have to iteratively add
data sources as the study progresses
 Surveys
Surveys are conducted when the use of a technique or tool already has taken place
(Pfleeger, 1994) or before it is introduced. It could be seen as a snapshot of the
situation to capture the current status. Surveys could, for example, be used for
opinion polls and market research.
When performing survey research the interest may be, for example, in studying how
a new development process has improved the developers attitudes towards quality
assurance or prioritizing quality attributes (Karlström et al., 2002). Then a sample of
developers is selected from all the developers at the company. A questionnaire is
constructed to obtain the information needed for the research.
The questionnaires are answered by the sample of developers. The information
collected are then arranged into a form that can be handled in a quantitative or
qualitative manner.
 Surveys
The purpose of survey is to understand the population, from which the sample
was drawn (Babbie, 1990). For example, by interviewing 25 developers on what
they think about a new process, the opinion of the larger population of 100
developers in the company can be assessed. Surveys aim at the development of
generalized conclusions.
Surveys have the ability to provide a large number of variables to evaluate, but it
is necessary to aim at obtaining the largest amount of understanding from the
fewest number of variables since this reduction also eases the data collection
and analysis work.
Surveys with many questions are tedious for respondents to fill out, and the data
quality may consequently decline.
 Surveys

The general objectives for conducting a survey is either of the following [6]:
Descriptive: Descriptive surveys can be conducted to enable assertions about some
population.
Explanatory: Explanatory surveys aim at making explanatory claims about the
population. For example, when studying how developers use a certain inspection
technique, we might want to explain why some developers prefer one technique
while others prefer another. By examining the relationships between different
candidate techniques and several explanatory variables, we may try to explain why
developers choose one of the techniques.
 Surveys

Explorative: Explorative surveys are used as a pre-study to a more thorough

investigation to assure that important issues are not foreseen. Creating a loosely
structured questionnaire and letting a sample from the population answer it could
do this. The information is gathered and analyzed, and the results are used to
improve the full investigation. In other words, the explorative survey does not
answer the basic research question, but it may provide new possibilities that could
be analyzed and should, therefore, be followed up in the more focused or thorough
survey.
 Focus Group

In interview-based data collection, the researcher normally asks a series of questions

to one person. In a focus group, data collection is conducted with several people at
the same time in a session resembling an interview.
This type of data collection technique is often very useful in qualitative research and
it has some strengths as listed by Kontio (Kontio et al, 2008):
• Discovery of new insights due to different backgrounds and the nature of the
group setting.
• Aided recall in the sense that people are able to confirm (or reject) facts that
would have been hidden in an individual interview.
 Focus Group

• Cost-efficiency due to the fact that several people are “interviewed” at the same
time.
• Depth due to the nature of a discussion compared to, for example, a
questionnaire.
• Business benefit to the participants since they are able to both network and
compare their own ways of working to the other participants.
 Focus Group
According to Robson (Robson, 2002) another advantage is that there is a natural
quality control in this kind of session. Participants tend to provide checks and react
to extreme views that they do not agree with, and group dynamics help in focusing
on the most important topics. There is also a chance that people who normally are
reluctant to be interviewed alone since they feel they have nothing to say may
participate in this kind of research.
There are weaknesses to focus groups, for example, if the group dynamics do not
work it may be hard to moderate the session. It requires that the moderator has
significant experience of both the subject area and moderating group discussions.
 Focus Group

It is also possible that secrecy inhibits participants from providing information that
they would provide in a normal interview. Here the researcher has to try to
anticipate if the questions concern material that may be sensitive in advance.
In some cases it is possible to formulate nondisclosure agreements between
participants before the meeting. However, if material are sensitive and secret,
writing a formal agreement may not solve this problem.
 Focus Group

People will maybe have “hidden agendas,” and emphasize answers for other reasons
than providing data. The researcher can try to anticipate the risk of this in advance
and try to monitor this during the session.
It should also be noted that there is limited time at a meeting and that it is not
possible to discuss too many topics and too complex topics in detail. This must be
considered in the planning phase of the focus group meeting.
 Focus Group
Example: Runeson conducted a study on unit testing practices across several
companies using focus group discussions. The focus group was composed of 17
representatives from 12 companies. The focus group discussions were organized
around three themes:
What is unit testing?
What are the participants?
Strengths regarding unit testing?
What are the participants?
Problems regarding unit testing?
 Interviews
Data collection through interviews is important in case studies. In interview-based
data collection, the researcher asks a series of questions to a set of subjects about
the areas of interest in the case study.
In most cases one interview is conducted with every single subject, but it is possible
to conduct group-interviews. The dialogue between the researcher and the
subject(s) is guided by a set of interview questions.
The interview questions are based on the topic of interest in the case study. The
interview questions are based on the formulated research questions (but they are of
course not formulated in the same way). Questions can be open, i.e. allowing and
inviting a broad range of answers and issues from the interviewed subject, or closed
offering a limited set of alternative answers.
Interviews
 Interviews

An interview session may be divided into a number of phases.

First the researcher presents the objectives of the interview and the case study, and
explains how the data from the interview will be used.
Then a set of introductory questions are asked about the background etc. of the
subject, which are relatively simple to answer.
After the introduction comes the main interview questions, which take up the
largest part of the interview.
 Interviews

If the interview contains personal and maybe sensitive questions, e.g. concerning
economy, opinions about colleagues, why things went wrong, or questions related to
the interviewees own competence (Hove and Anda 2005), special care must be
taken. In this situation it is important that the interviewee is ensured confidentiality
and that the interviewee trusts the interviewer. It is not recommended to start the
interview with these questions or to introduce them before a climate of trust has
been obtained. It is recommended that the major findings are summarized by the
researcher towards the end of the interview, in order to get feedback and avoid
misunderstandings.
 Interviews

The three general principles of Interview sessions by Caroline Seaman, personal communication
 Interviews
During the interview sessions, it is recommended to record the discussion in a
suitable audio or video format. Even if notes are taken, it is in many cases hard to
record all details, and it is impossible to know what is important to record during the
interview (Hove and Anda 2005).
When the interview has been recorded it needs to be transcribed into text before it
is analyzed. This is a time consuming task, but in many cases new insights are made
during the transcription, and it is therefore not recommended that this task is
conducted by anyone else than the researcher.
In some cases it may be advantageous to have the transcripts reviewed by the
interview subject. In this way questions about what was actually said can be sorted
out.
 Interviews

During the planning phase of an interview study it is decided whom to interview.

Due to the qualitative nature of the case study it is recommended to select subjects
based on differences instead of trying to replicate similarities. This means that it is
good to try to involve different roles, personalities, etc in the interview. The number
of interviewees has to be decided during the study. One criterion for when sufficient
interviews are conducted is “saturation”, i.e. when no new information or viewpoint
is gained from new subjects (Corbin and Strauss 2008).
 Observations

Observations can be conducted in order to investigate how a certain task is

conducted by software engineers. There are many different approaches for
observation.
One approach is to monitor a group of software engineers with a video recorder and
later on analyze the recording, for example through protocol analysis (Owen et al.
2006; von Mayrhauser and Vans 1996).
 Observations

Another alternative is to apply a “think aloud” protocol, where the researcher are
repeatedly asking questions like “What is your strategy?” and “What are you
thinking?” to remind the subjects to think aloud. This can be combined with
recording of audio and keystrokes as proposed e.g. by Wallace et al. (2002).
Observations in meetings is another type, where meeting attendants interact with
each other, and thus generate information about the studied object.
 Observations
Approaches for observations can be divided into high or low interaction of the
researcher and high or low awareness of the subjects of being observed