Science of Computer Programming 135 (2017) 75–87

Contents lists available at ScienceDirect

Science of Computer Programming

www.elsevier.com/locate/scico

Benchmarking IT operations cost based on working time and

unit cost
Masateru Tsunoda a,b,∗ , Akito Monden c , Kenichi Matsumoto a , Sawako Ohiwa d ,
Tomoki Oshino d
a
Graduate School of Information Science, Nara Institute of Science and Technology, 8916-5 Takayama, Ikoma, Nara 630-0192, Japan
b
Department of Informatics, Kindai University, 3-4-1 Kowakae, Higashi-Osaka, Osaka 577-8502, Japan
c
Graduate School of Natural Science and Technology, Okayama University, 1-1-1 Tsushima-Naka, Okayama-shi, Okayama 700-8530, Japan
d
Economic Research Institute, Economic Research Association, Ryoshin-onarimon Building, 6-17-15 Shinbashi, Minato-ku, Tokyo 105-0004,
Japan

a r t i c l e i n f o a b s t r a c t

Article history: Recently, size of information system gets large, and the information system operation (IT
Received 15 March 2016 operations) is often outsourced. When IT operations of the large system is outsourced,
Received in revised form 23 October 2016 high cost is needed, and troubles in IT operations may affect the activity of the company.
Accepted 25 October 2016
Therefore, the information system operation is important for the companies. Cost is one of
Available online 4 November 2016
the important factors when the system operation is outsourced. However, it is not easy
Keywords: for the customers (system users) to judge whether the operation cost is valid or not.
Benchmarking To support the judgment, we focus on information which the customers can know (e.g.,
IT operations size of software), to estimate the working time. In the analysis, we clarified the factors
Work efficiency which affect the working time. Then, data was stratified based on the factors, to show
Regression analysis the benchmark of working time. Using the benchmark, customers estimate the working
time roughly. Also, we clarified the factors which affect the unit cost, and showed the
benchmark. Operation cost can be estimated, by estimated working time multiplied by the
estimated unit cost. The analysis results showed that the process standardization relates
to the working time of operators. Also, the network range and the contract type have
a relationship to the unit cost of operators. Work efficiency and unit cost do not affect
operation quality.
© 2016 Elsevier B.V. All rights reserved.

1. Introduction

In recent years, the size of information systems has become large, and the operation of these information systems (IT
operations) is often outsourced. Given this scenario, the operation of an information system, which consists of computers,
a network, and software, is important. The operation of an information system involves the management of computers and
the network by operators, e.g., they replace updated software and recover the system when system failure occurs. Operation
of the system has resulted in the standardization of system operation processes, resulting in specifications such as the ITIL
(Information Technology Infrastructure Library) [1] and ISO20000 receiving more attention.
Cost is one of the important factors to consider when the operation of a system is outsourced. However, it is not easy for
the customers (system users) to judge whether the operation cost is realistic. This is because, generally, customers do not

* Corresponding author to: Department of Informatics, Kindai University, 3-4-1 Kowakae, Higashi-Osaka, Osaka 577-8502, Japan.
E-mail address: tsunoda@info.kindai.ac.jp (M. Tsunoda).

http://dx.doi.org/10.1016/j.scico.2016.10.006
0167-6423/© 2016 Elsevier B.V. All rights reserved.
76 M. Tsunoda et al. / Science of Computer Programming 135 (2017) 75–87

know the operation cost in other customers’ companies, and therefore they cannot compare the cost with others to judge
the validity of the cost. The goal of our study is to provide customers with support to enable them to assess the validity of
the cost.
Generally, the relationship between the operational cost and working time of the operation service supplier (outsourced
operators) is considered to be strong. Therefore, if the working time of the supplier is known, the operational cost can
be estimated, and the estimated cost can be used to judge the validity of the actual cost. However, it is not easy for
the customers to obtain the working time of the supplier. Thus, we focus on information of which the customers have
knowledge (e.g., the size of software), to estimate the working time. In the analysis, first, we clarified the factors that affect
the working time. Then, data was stratified based on these factors, to use as a benchmark for the working time. This would
enable customers to use the benchmark to roughly estimate the working time.
Knowing the estimated working time only is insufficient to estimate the operational cost. The unit cost of outsourced
operators is also needed. That is, the operation cost can be estimated by using the estimated working time multiplied by
the estimated unit cost. We obtained a rough estimation of the unit cost by clarifying the factors that affect the unit cost,
and determined a benchmark of the unit cost, stratifying the dataset based on the factors.
The major contribution of the study is benchmarking the operation cost of the system (NOT increasing the work efficiency
of system operation). Although using the work efficiency as benchmark is useful for service suppliers, it is not utilized to
enable the customers to judge the validity of the cost. In contrast, using the cost as benchmark is useful for performing
judgment. To the knowledge of the authors, no study in which the system operational cost for customers was analyzed has
been reported to date.
In our previous study [18], we analyzed the factors that affect the system operational cost. In the study, we regarded
the cost to be derived from the number of operators, the number of computers, and the unit cost of outsourced operators.
This is because, in the dataset used in [18], the working time was not recorded in most data points. The scope of this
work is different from that of the previous work in the sense that, previously, the number of operators was used, instead of
the time. Note that when the number of operators is regarded as working time, the analysis is approximate. For example,
when the number of operators is four and they each work two hours a day, the total working time in a day is eight hours,
whereas when the number of operators is two and they each work four hours a day, the total working time in a day is also
eight hours.

2. Related work

Although some studies considering system operation have been published, they mainly focused on the process of the
system operation. For instance, Brooks [4] determined metrics for evaluation of the system operation process. Pollard et
al. [15] performed a case study of system operation of companies in the United States and Australia, and identified some
critical success factors (CSFs) for successful ITIL implementations. The research also focused on the system operation process.
However, these studies cannot be used to benchmark the system operation.
Some studies focused on the operational cost of the system. Brown et al. [5] analyzed whether automation decreases
the operational cost. Han et al. [8] analyzed the relationship between IT outsourcing and non-IT operational cost. However,
they did not analyze the factors affecting the operational cost. Mannino et al. [13] analyzed the efficiency of data warehouse
operations by carrying out a data envelopment analysis (DEA). Although data warehouse operations are one part of the
system operation, their results cannot be applied to benchmarking. No study involving an estimation of quantitative system
operation staff (cost) has been reported. The ITIL explains cost estimation based on cost types such as equipment cost or
organization (staff) cost [3]. However, it does not explain how to estimate the number of staff quantitatively.
Selected articles present varying extents of benchmarking of the system operation. For, example, some researchers [6]
determined the number of companies that use tools to operate the system, and so on. Other authors [9] present summarized
data such as the number of operation staff. In addition, Potter et al. [16] established a benchmark of cost metrics such as
the ratio of IT spending to revenue. However, they did not analyze factors affecting the unit cost of staff and working time.
Thus, it is difficult for customers to use the information in these articles to validate the operational cost.

3. Research methodology

3.1. Overview

Our research methodology involves collecting data of IT operations from companies, and to analyze the variables statis-
tically to validate assumptions about the cost. The detail of the data collection procedure is the following:

1. Contrive candidates of factors affecting the activities of IT operations.

2. Define variables to measure the factors, and formulate questions to collect the variables.
3. Send out the questionnaire containing the questions.
4. Collect the answers from the completed questionnaires, and collate them.
5. Eliminate variables for which many values are missing, and select variables considered to relate to the operational cost.
6. Analyze the variables statistically to validate assumptions about the cost.
 M. Tsunoda et al. / Science of Computer Programming 135 (2017) 75–87 77

Table 1
Description of variables.

No. Variable (unit) Description

V1 Operation cost (Japanese yen) Total cost required for the operation in a year. Cost means the contract price, rather than the real
cost to the service supplier.
V2 Total working time (hour) Working time needed to operate the system in a year. It is defined as working time of the
customers + working time of the service supplier.
V3 Working time of service supplier (hour) Working time of the service supplier needed to operate the system in a year.
V4 Disruption time (hour) Sum of disruption time of the operated system due to failures in a year.
V5 Work efficiency Work efficiency of the IT operations. It is defined as number of programs/total working time.
V6 Unit cost of the operators Unit cost of the operators of the system. It is defined as operation cost/working time of service
supplier.

V7 Maximum number of users Maximum number of users on the operated system.

V8 Number of programs Number of programs which consist of the operated system.

V9 Process standardization Extent of process standardization of the operated system. U1: process is standardized, U2: process
standardization is work in progress, and U3: process is not standardized.
V10 Social impact Extent of social impact of the operated system. C1: social impact of the system is almost negligible,
C2: system has social impact, but it is limited, and C3: social impact is very large.
V11 Network range Network range of the operated system. N1: network is between customers and other organizations,
N2: network is between customers and other branch offices, N3: it is within a branch office, N4:
network is within a department, and N5: system does not connect to other systems.
V12 SLA status Status of SLA of the operated system. S1: SLA exists (goals are guaranteed), S2: SLA exists (goals are
not guaranteed), and S3: SLA does not exist.
V13 Contract type Contract type of the operated system. K1: dispatched workers, K2: quasi-mandate workers, K3:
contract workers, K4: combined use of quasi-mandate and contract workers, and K5: others.

In the next subsection, we explain the dataset collected with the above procedure.

3.2. Dataset

The dataset used in the study was collected in 2011 and 2013 by the Economic Research Association. It includes 179
data points collected from both customers and operation service suppliers. The size of the systems recorded in the dataset
varies (i.e., data from both small systems and large systems are included). This is expected to improve the generality of the
analysis results derived from the dataset to some extent.
The dataset includes variables that relate to activities associated with IT operations. From these variables, we selected
variables by considering the characteristics of IT operations, the analysis result of our previous study [18], and factors
affecting software development [19]. Table 1 lists the variables used in the analysis. Some values were missing from the
set of values obtained for each of the variables. We removed these data points by list-wise [11] deletion in the regression
analysis (List-wise deletion removes data points (projects) of which the variables have one or more missing values). Thus, the
number of data points was different in each analysis. The following list contains additional explanations for the categorical
variables.

• V9: The standard procedure of the operation is sometimes defined, to enhance the efficiency of the operation and to
suppress system problems. The extent is signified by three levels.
• V10: Some information systems are used as infrastructure of society. If the system experiences problems, many social
activities are affected. We refer to the influence as the social impact of the system. The social impact can occur on three
levels.
• V11: Information system sometimes connects to other systems using the network. The network range is signified by
five levels.
• V12: When the system operation is outsourced, an SLA (Service Level Agreement) is sometimes concluded. The SLA
assures the service level of the operation, such as the system recovery time, in advance. The SLA status denotes the SLA
of each data point.
• V13: There are different contract types of the system operation, where the contract type refers to the working type of
the operators. There are four types of contracts.

In the table, variables V1...V6 are outcomes of IT operations such as the cost and the working time, variables V7 and V8
signify system size, and variables V9...V13 are candidates of factors affecting the outcomes.
Categorical variables such as V10 are recorded based on respondents’ self-assessment. This method is the same as the
cost drivers of COCOMO [2]. To suppress the variance of the assessment, we presented a detailed definition of the items to
respondents. For example, the definition of C3 is that “The system is the infrastructure of social and economic activities. If
part of or none of the functions are available, the influence on social and economic activities is very large. An example of
the system is an infrastructure system used by the general public.”
78 M. Tsunoda et al. / Science of Computer Programming 135 (2017) 75–87

Fig. 1. Relationship between the operation cost and service supplier’s working time per year.

3.3. Statistical analysis methods

To analyze the relationship between two variables, we used regression analysis, in which we focused on the Adjusted
R2 and standardized partial regression coefficients. Before building the model, all variables of the ratio scale were log-
transformed. Dummy variables were created when categorical variables such as V9 were used as independent variables. We
applied stepwise variable selection to build the model. The variables were included in the model when the p-value was
smaller than 0.1, and removed when the p-value was larger than 0.2.
If the significance level is set too high, the probability of a type I error increases. In contrast, when the significance level
is set too low, the probability of a type II error increases. In our dataset, the number of data points was not very large. For
example, the number of data points that was used in the regression analysis was about 60. Considering the probability of
type I and type II errors, we set the significance level to 10%, which is the same as in other software engineering studies
[1,10,17].
Furthermore, we used scatter plots and boxplots to visualize the data. In the boxplots, the bold line in each box indicates
the median value. Small circles indicate outliers, that is, values that are more than 1.5 times larger than the 25%–75% range
from the top of the box edge. Stars indicate extreme outliers, whose values are more than 3.0 times larger than this range.
To enhance the visibility of the boxplots, some outliers were omitted.

4. Relationship between operation cost and working time

In the analysis, we clarified the factors that determine the system operation cost, which is the total cost required to
perform the operation in a year. Note that the cost refers to the contract price, rather than the real cost to the service
supplier. Most of the system operation cost is considered to be based on the labor cost of the operators. In other words,
the cost is assumed to relate strongly to the working time of the operators of the service supplier. Note that the system is
operated by not only the service supplier, but also the customer. Therefore, the working time of the service supplier should
be distinguished from the working time of the customer, when analyzing the operation cost. We clarified the purpose of
the analysis by posing the following research question.

RQ1: Does the operation cost depend on the working time of the service supplier?

We analyzed the relationship between the operation cost and the working time of the service supplier, using regression
analysis, in which the dependent and independent variables were the operation cost, and the working time of the service
supplier, respectively. As a result, the built model of R2 was found to have a value of 0.91. This means the operation cost
mostly depends on the labor cost. In other words, if the customers know the working time of the service supplier, the
operation cost can be estimated.
As explained above, the cost is the contract price, and the price includes the profit and expenses of the service supplier.
They are one of the factors that affect the operation cost. Moreover, the unit cost of the operators of the service supplier is
one of the factors. However, their influence is not very strong, since R2 of the model was very large (0.91).
Fig. 1 shows the relationship between the operation cost and the working time of the service supplier. To avoid abusing
the result, we do not show the operation cost on the vertical axis. Although there are two outliers in the upper left of the
figure (their cost is higher than others, considering the working time), the relationship between the cost and the time was
strong for other data points. This result is not always natural in the software engineering field. For example, in software
maintenance activities, the relationship between the maintenance cost and the working time was relatively weak [19].
Based on the analysis, the answer to RQ1 is:

“Yes. The operation cost mostly depends on the working time of the service supplier.”
 M. Tsunoda et al. / Science of Computer Programming 135 (2017) 75–87 79

Table 2
Regression model using system size.

Variable Std. partial reg. coefficient p-value

Maximum number of users 0.41 0.0%
Number of programs 0.49 0.0%

The result also suggests that the service suppliers settle the contract price properly, i.e., they set the price based on the
labor cost of the operators. However, in a few cases, the price was relatively very high, compared with the labor cost. In
contrast, there was no case in which the price was very low, compared with the cost, although this would naturally be
expected.
Especially from the viewpoint of service suppliers, the cost differs from the price, because the price is the sum of cost
and profit. However, we analyzed the dataset from the viewpoint of customers. From this viewpoint, cost and price do
not differ significantly. Additionally, as shown above, the relationship between price and cost (i.e., the working time of the
service supplier) is very strong. Therefore, we refer to the price as the cost in the paper.

5. Factors affecting working time

Hereinafter, the sum of the working time of the customers and the service supplier in a year is referred to as the total
working time. As explained in section 4, the information system is operated by the customers, not only by the service
suppliers. The ratio of the operators of the customers to the service suppliers was not the same for each data point. Thus,
using the working time of the service supplier is not appropriate in the analysis. Instead, we used the total working time in
the following analysis.
We benchmarked the total working time by defining the work efficiency as the number of programs divided by the
total working time. In the next subsection, we explain why the number of programs is used in the definition. In the
following analysis, we show the relationship between the work efficiency and other factors using boxplots. We obtained
an approximate estimation of the working time by requiring customers to divide the number of programs by the work
efficiency shown in the boxplots (Additional details can be found in section 7.2).
We used multiple regression analysis to analyze factors affecting the total working time. In the model, the dependent
variable was the total working time. Note that although the contract type was analyzed in section 6, it was not analyzed in
this section, because an association between the contract type and working time is not considered to exist.

5.1. Relationship to system size

The purpose of the analysis in the subsection is to answer the research question.
RQ2: What does the total working time depend on?
As explained in section 1, the main activities of the system operation is to manage the computers and network in
the system. Thus, the system size such as the software size and the number of computers are considered to be the main
factor determining the total working time. We obtained the software size by using the number of programs, instead of
the functional size of the software on the system, because the number of values missing from the number of programs
was relatively small. The dataset includes various variables such as the number of Windows servers and the number of
mainframes. Based on the preliminary analysis, we used the number of Windows servers as the hardware size of the
system.
When the number of users increases, the response time to the users may also increase; therefore, we also analyzed the
maximum number of users on the system. Using multiple linear regression analysis, we analyzed the influence of these
factors on the total working time. In the model, the independent variables were the number of programs, the number of
Windows servers, and the maximum number of users. We applied variable selection to build the regression model.
Table 2 provides the standardized partial regression coefficients in the built model. The number of Windows servers
was removed from the model by using variable selection. This means the number of Windows servers is not indispensable
to estimate the total working time. As the partial regression coefficient of the number of programs was larger than the
maximum number of users, we used the number of programs to define the work efficiency. Note that the latter number
was not small, and hence the influence of the maximum number of users was not negligible. Hence, we also used the
maximum number of the users to build regression models in the following. Thus, the answer to RQ2 is:

“The total working time depends on the number of programs and the maximum number of the users to some extent.”

The Adjusted value of R2 was 0.53, which means that the total working time is explained by the number of programs
and the maximum number of users, since the Adjusted R2 was larger than 0.5. Fig. 2 and Fig. 3 show the relationships
between the total working time and the variables. In the figures, data points are scattered, suggesting that the estimation is
difficult when using the number of programs or the maximum number of users only.
Therefore, in the following analysis, we focused on other factors considered to affect the total working time.
80 M. Tsunoda et al. / Science of Computer Programming 135 (2017) 75–87

Fig. 2. Relationship between the maximum number of users and service supplier’s working time.

Fig. 3. Relationship between the number of programs and service supplier’s working time.

Table 3
Regression model using process standardization.

Variable Std. partial reg. coefficient p-value

Maximum number of users 0.47 0.0%
Number of programs 0.47 0.0%
Process standardization −0.16 9.3%

5.2. Relationship to process standardization

We assumed that when the operation process is standardized, the work efficiency is improved, and it affects the total
working time. Thus, we set the following research question:

RQ3.1: Does process standardization relate to work efficiency?

We analyzed the relationship between process standardization and the total working time using multiple regression
analysis. In the model, the independent variables were the number of programs, the maximum number of users, and the
extent of process standardization. We created a dummy variable that denotes whether the process standardization is level
U1, because the distribution of U1 differed from that of the other levels (see Fig. 4).
Standardized partial regression coefficients of the model are presented in Table 3. The Adjusted R2 of the model was
0.54, which is very similar to the model without the process standardization built in section 5.1, despite the fact that the
regression coefficients of the process standardization are not very small. The p-value of the partial regression coefficient
of the process standardization was smaller than 10%. Therefore, the process standardization is related to the total working
time. The regression coefficient of the process standardization was negative; thus, when the process is standardized (i.e.,
level U1), the total working time decreases, and work efficiency improves greatly.
Fig. 4 shows the relationship between the work efficiency and process standardization. The plot in the figure contains
many data points of which the work efficiency was high in level U1, although the variance of level U1 was large. Based on
the boxplot and the regression analysis, the answer to RQ3.1 is:

“The process standardization is regarded to relate to the work efficiency.”

 M. Tsunoda et al. / Science of Computer Programming 135 (2017) 75–87 81

Fig. 4. Relationship between process standardization and work efficiency.

Table 4
Regression model using social impact.

Variable Std. partial reg. coefficient p-value

Maximum number of users 0.37 0.0%
Number of programs 0.44 0.0%
Social impact −0.16 10.6%

Fig. 5. Relationship between social impact and work efficiency.

5.3. Relationship to social impact

When an information system has a high social impact, the system should be operated carefully. It may need more
working time than a system with a small social impact. Thus, we set the following research question:

RQ3.2: Does the social impact relate to work efficiency?

We analyze the influence of social impact on the total working time. In the multiple regression analysis, the independent
variables were the number of programs, the maximum number of users, and the extent of the social impact of the system.
A dummy variable denotes whether the social impact is at level C1, because the value of C3, the number of data points, was
small.
Table 4 lists the standardized partial regression coefficients of the model. The Adjusted R2 of the model was 0.54, and it
was almost the same as the model built in section 5.1. However, the regression coefficients of the process standardization
were not very small. Although the p-value of the coefficient of the social impact was relatively small, it was larger than
10%. Therefore, the answer to RQ3.2 is:

“It is probable that the social impact relates to the total working time.”

The regression coefficient of the social impact was negative. Hence, when the social impact is small (i.e., level C1), the total
working time decreases, and the work efficiency increases.
The relationship between the work efficiency and the social impact is shown in Fig. 5. As the number of data points
at level C3 was two, we disregard this level. Although the median of level C1 and C2 is almost the same, the variance of
C1 was large, and at level C2, there were few data points of which the work efficiency was high. That is, when the social
impact is not small, the work efficiency tends to be lower.
82 M. Tsunoda et al. / Science of Computer Programming 135 (2017) 75–87

5.4. Relationships to network range and SLA

When the network is complex, the system operation may need more working time, compared with similar sized systems.
Thus, we set the following research question:

RQ3.3: Does the network range relate to work efficiency?

We analyzed the relationship between the network range and the total working time. In the multiple regression analysis,
the independent variables were the number of programs, the maximum number of users, and the network range. Section 6.2
describes how we handled the network range. The network range was removed from the model by variable selection, and
the p-value of the regression coefficient was 0.97. Therefore, the answer to RQ3.3 is:

“The network range is not related to the total working time.”

Although an SLA ensures quality of the system operation, it may also increase the amount of working time required.
Thus, we set the following research question:

RQ3.4: Does the SLA status relate to work efficiency?

We applied multiple regression analysis to clarify the influence of the SLA on the total working time. The independent
variables were the number of programs, the maximum number of users, and the SLA status. Variable selection did not result
in SLA status being chosen, and the p-value of the regression coefficient was 0.21. Thus, the answer to RQ3.4 is:

“The SLA status is not explicitly related to the total working time.”

5.5. Analysis of working time considering multiple factors

If the process standardization is related to the social impact, one of these would strongly relate to the work efficiency,
but the other may not be related very much. Thus, we set the following research question:

RQ3.5: What are the major factors affecting the work efficiency?

To identify the major factors affecting the total working time, we built a multiple regression model, in which the inde-
pendent variables were the number of programs, the maximum number of users, the process standardization, and the social
impact.
Variable selection removed the social impact from the model. This means the total working time is explained by the
number of programs, the maximum number of users, and the process standardization. Thus, the relationship between the
total working time and the social impact was weaker than other relationships, and the answer to RQ3.5 is:

“The major factor is the process standardization.”

Note that we do not show the built model because it is same as the model built in section 5.2.

6. Factors affecting unit cost

An estimation that only includes the working time is insufficient to estimate the operation cost. The unit cost of out-
sourced operators is also required, as explained in Section 1. We defined the unit cost of operators as the total cost divided
by the working time of the service supplier. If there is a factor that affects the operation cost, in addition to the working
time, the factor is considered to affect the unit cost. Therefore, we analyzed the factors affecting the operation cost by using
multiple regression analysis. In the model, the dependent variable was the operation cost.

6.1. Relationships to process standardization and social impact

We assumed that a standardized operation process would result in a complex system and would require highly skilled
operators. This may increase the unit cost of operators. Thus, we set the following research question:

RQ4.1: Does process standardization relate to unit cost?

We analyzed the relationship between process standardization and the operation cost. In the multiple regression analysis,
the independent variables were the working time of the service supplier and process standardization. Process standardiza-
tion was removed from the model by variable selection, and the p-value of the regression coefficient was 0.17. Therefore,
the answer to RQ4.1 is

“Process standardization does not have the relationship to the operation cost.”
 M. Tsunoda et al. / Science of Computer Programming 135 (2017) 75–87 83

Table 5
Regression model using network range.

Variable Std. partial reg. coefficient p-value

Working time of service supplier 0.95 0.0%
Network range −0.12 2.0%

Fig. 6. Relationship between network range and unit cost.

When an information system has a high social impact, highly skilled operators may be needed. This can be expected to
enhance the unit cost of operators. Thus, we set the following research question:

RQ4.2: Does the social impact relate to unit cost?

We applied multiple regression analysis to clarify the influence of the social impact on the operation cost. The inde-
pendent variables were the working time of the service supplier and the social impact. Variable selection resulted in the
elimination of social impact, and the p-value of the regression coefficient was 0.29. Thus, the answer to RQ4.2 is:

“The social impact status does not have an explicit relationship to the operation cost.”

6.2. Relationship to network range

When the network is complex, system operators who have extensive network skills are needed to operate the system,
and this may increase the unit cost of operators. Thus, we set the following research question:

RQ4.3: Does the social impact relate to unit cost?

We analyzed the relationship between the network range and the operation cost, i.e., the influence of the network range
on the unit cost. In the multiple regression analysis, the independent variables were the working time of the service supplier
and the network range. We used the network range as the ordinal scale, considering the relationship between the unit cost
and the network range (see Fig. 6). We replace level N5 as level N3, since the number of the data points was one, and there
is no data point with a network range at level N4.
The standardized partial regression coefficients of the model are shown in Table 5. The Adjusted R2 of the model was
0.92. The p-value of the partial regression coefficient of the process standardization was smaller than 5%. Therefore, the
answer to RQ4.3 is

“The network range is related to the operation cost.”

The regression coefficient of the process standardization was negative. Therefore, when the network range is narrow (e.g.,
level N3), the operation cost decreases, and the unit cost becomes low. Fig. 6 shows the relationship between the network
range and the unit cost. As explained above, the number of the data points of level N1 was one, and there is no data point
of which the network range was at level N4. The currency unit of the vertical axis is the Japanese yen. The figure also shows
that when the network range is narrower, the unit cost is lower.

6.3. Relationship to contract types

When the contract type is different, the unit cost of the operators may also be different. Thus, we set the following
research question:

RQ4.4: Does the contract type relate to unit cost?

84 M. Tsunoda et al. / Science of Computer Programming 135 (2017) 75–87

Table 6
Regression model using contract type.

Variable Std. partial reg. coefficient p-value

Working time of service supplier 0.97 0.0%
Contract type K1 −0.07 19.1%
Contract type K2 −0.01 87.9%
Contract type K3 −0.12 4.5%

Fig. 7. Relationship between contract type and unit cost.

Table 7
Regression model using SLA.

Variable Std. partial reg. coefficient p-value

Working time of service supplier 0.93 0.0%
SLA status −0.09 9.7%

To clarify the relationship between the contract type and the operation cost, we applied multiple regression analysis, in
which the independent variables were the working time of the service supplier and the contract type. The dummy variables
of type K5 were not used, because there was only one data point. Note that we did not create dummy variables of type K4,
since it can be denoted by the variables of type K1, K2, and K3.
Table 6 shows the standardized partial regression coefficients of the model. The Adjusted R2 of the model was 0.91. The
p-value of the coefficient of the contract type (type K3) was smaller than 5%. Therefore, the answer to RQ4.4 is:

“The contract type of K3 is related to the operation cost.”

The regression coefficient of the value was negative. Hence, when the contract type is K3, the operation cost decreases,
and the unit cost also becomes low. The relationship between the operation cost and the contract type is shown in Fig. 7.
Types K1 and K5 only had a few data points each; thus, we disregard these two types. The figure shows that type K3 has
the lowest unit cost.

6.4. Relationship to SLA

SLA ensures quality of the system operation; however, it may also enhance the unit cost of the operators. Thus, we set
the following research question:

RQ4.5: Does the SLA status relate to unit cost?

We applied multiple regression analysis to clarify the influence of SLA on the operation cost (i.e., the unit cost). The
independent variables were the working time of the service supplier and SLA status.
The standardized partial regression coefficients of the model are shown in Table 7. The Adjusted R2 of the model was
0.91. The p-value of the coefficient of the SLA status (status S1) was smaller than 10%. The regression coefficient was
positive. This means when an SLA is agreed upon (i.e., status S1), the operation cost increases, and the unit cost becomes
high. Fig. 8 shows the relationship between the SLA status and the unit cost. In the figure, although the median of status
S3 was lower than that of the others, the distribution was not very different. That is, the answer to RQ4.5 is:

“The relationship between the SLA status and the unit cost was not explicit.”
 M. Tsunoda et al. / Science of Computer Programming 135 (2017) 75–87 85

Fig. 8. Relationship between SLA and unit cost.

Table 8
Regression model using newwork range and contract type.

Variable Std. partial reg. coefficient p-value

Working time of service supplier 0.97 0.0%
Network range −0.10 4.6%
Contract type K3 −0.09 5.6%

6.5. Analysis of operation cost considering multiple factors

There may be relationships between the network range, contract type, and SLA status. If there are relationships, this
factor would be expected to strongly affect the operation cost, but the other factors may not affect it very much. Thus, we
set the following research question:

RQ4.6: What are the major factors of the operation cost?

We clarified the major factors by applying multiple regression analysis. In the analysis, the independent variables were
the working time of the service supplier, the network range, the contract type, and SLA status. We created dummy variables
to denote status S1 and S2.
Table 8 presents the standardized partial regression coefficients of the model. Variable selection was applied to build
the model, and SLA status was removed from the model. Thus, the relationship between the operation cost and SLA status
was found to be weaker than the other relationships. On the contrary, the network range and the contract type cannot be
ignored when estimating the unit cost. Hence, the answer to RQ4.6 is:

“The major factors are the network range and the contract type.”

7. Discussion

7.1. Relationships between quality and cost

In terms of system operation, high work efficiency and low unit cost of operators are preferable, when focusing on cost
only. However, if these two factors affect the quality of the operation negatively, they are not preferable on the whole. Thus,
we analyzed the relationships between quality, work efficiency, and unit cost. We regarded disruption time (see Table 1)
as contributing to the quality of the system operation. We used Spearman’s rank correlation coefficient to suppress the
influence of outliers.
The result is shown in Table 9. Both work efficiency and unit cost had a slight negative correlation to disruption time.
This means when both the work efficiency and unit cost are high, the disruption time becomes shorter. However, the
correlation coefficient was small, and it was not statistically significant. Hence, the work efficiency and unit cost are not
considered to affect the operation quality explicitly.

7.2. Applying benchmark to practical use

Here we illustrate how to apply the benchmark in practice. Customers estimate the approximate operation cost as fol-
lows:

1. Count the number of programs on the system.

2. Divide the number of programs by the work efficiency shown in the boxplots, to estimate the total working time.
86 M. Tsunoda et al. / Science of Computer Programming 135 (2017) 75–87

Table 9
Relationships between quality, work efficiency and unit cost.

Work efficiency Unit cost

Correlation coefficient −0.19 −0.15
p-value 29% 52%
# of data points 33 21

3. Subtract the working time of the customer from the estimated total working time, to estimate the working time of the
service supplier.
4. Multiply the estimated working time of the service supplier by the unit cost shown in the boxplots, to estimate the
operation cost.

In step 2, if the social impact of the system is C2, the work efficiency is regarded as about 0.2, for example. The es-
timation accuracy can be enhanced by selecting a smaller box. For instance, when the process standardization is U1, and
the social impact on the system is C2, the work efficiency should be decided based on social impact, because the size of
the box of C2 is smaller than that of U1. Likewise, the unit cost is decided in step 4. Although the estimated cost is an
approximation, it is useful to validate the operation cost.
Note that providing a more formal estimation model would be more appropriate. However, the models built in section 5
did not have sufficient explanatory power, since their R2 values were not very large. Therefore, we did not present the
model, to avoid abusing the result. Although the boxplots shown in the paper do not indicate the estimated cost explicitly,
they are useful to show the variance of the estimation. For example, when process standardization is work in progress, the
box U2 is not very high as shown in Fig. 4, and the estimation was somewhat reliable since the variance of work efficiency
is small. In contrast, the variance of U1 is large, and hence the estimation using the boxplot is not very reliable.

7.3. Threats to validity

Internal validity: As explained in section 3.2, data from both small and large systems were included in the dataset.
When we regard a small system to have an operation cost less than 100 million yen, the percentage of data points is about
30%. Similarly, when we regard a large system to have an operation cost in excess of 1,000 million yen, the percentage of
the data points is about 20%. Thus, the influence of selecting a dataset is considered to be small.
External validity: In the analysis of software development productivity (work efficiency), the factors affecting produc-
tivity are similar between countries [12,14,20,21] (e.g., business sector and programming language). Thus, the relationship
clarified in this paper is considered to be observed in other markets. However, Cusumano et al. [7] showed that the pro-
ductivity is different between countries. Thus, the work efficiency shown in the boxplots should not be regarded as being
absolutely true in other markets.
Construct validity: A function point is a more appropriate measure of software size. However, compared with software
development, the number of programs is not highly inappropriate as a measure of software size in IT operations. IT op-
erations do not perform software development, and hence the software size does not increase the working time linearly.
In contrast, increasing the number of programs may increase the working time, even when the software size does not
change, because operations such as release management and configuration management are affected by the number of pro-
grams. Note that the relationship between the number of programs and the functional size was very strong (the correlation
coefficient was 0.80, the p-value was 0%, and the number of data points was 18).
Not all the companies use functional size measurement; thus, many values of functional size are missing from the
dataset. This means it is difficult for service suppliers and customers (i.e., non-developers) to know the functional size. If
the functional size is used in the analysis, it is difficult to apply the benchmarking procedure explained in the previous
subsection.

8. Conclusions

Our goal is to provide support for customers to enable them to validate the operation cost of an information system.
To achieve the goal, we clarified the factors that affect the working time and the unit cost of the operators, using a cross-
company dataset. The outcome of the analysis was that the operation cost has a strong relationship to the working time
of operators, and the total working time is settled based on the number of programs and maximum number of users. The
working time of operators was found to be related to the process standardization. The unit cost of operators has a relation-
ship to the network range and the contract type. We stratified the dataset based on these factors, to obtain an approximate
estimation of the working time and unit cost. Work efficiency and unit cost were determined not to affect operation quality
explicitly.
 M. Tsunoda et al. / Science of Computer Programming 135 (2017) 75–87 87

Acknowledgements

This research was partially supported by the Japan Ministry of Education, Culture, Sports, Science, and Technology [Grant-
in-Aid for Scientific Research (C) (No. 25330090)].

References

[1] J. Blackburn, G. Scudder, L. Wassenhove, Improving speed and productivity of software development: a global survey of software developers, IEEE
Trans. Softw. Eng. 22 (12) (1996) 875–885.
[2] B. Boehm, Software Engineering Economics, Prentice Hall, 1981.
[3] J. Bon (Ed.), Foundations of IT Service Management: Based on ITIL, Van Haren Publishing, 2005.
[4] P. Brooks, Metrics for IT Service Management, Van Haren Publishing, 2006.
[5] A. Brown, J. Hellerstein, Reducing the cost of IT operations: is automation always the answer?, in: Proc. of Conference on Hot Topics in Operating
Systems, HOTOS, vol. 10, 2005, pp. 12–22.
[6] Continuity Software, IT operations analytics benchmark survey, http://www.continuitysoftware.com/wp-content/uploads/2013/12/2013-ITOA-survey.pdf,
2013.
[7] M. Cusumano, A. MacCormack, C. Kemerer, B. Crandall, Software development worldwide: the state of the practice, IEEE Softw. 20 (6) (2003) 28–34.
[8] K. Han, S. Mithas, Information technology outsourcing and non-IT operating costs: an empirical investigation, MIS Q. 37 (1) (2013) 315–331.
[9] Kaseya, IT operations benchmark survey 2015, http://www.kaseya.com/download/en-US/white_papers/WhitePaper.2015MMEMaturitySurvey.Kaseya.pdf,
2015.
[10] N. Lee, C. Litecky, An empirical study of software reuse with special attention to ADA, IEEE Trans. Softw. Eng. 23 (9) (1997) 537–549.
[11] R. Little, D. Rubin, Statistical Analysis with Missing Data, 2nd edition, Wiley, 2002, p. 408.
[12] C. Lokan, T. Wright, P. Hill, M. Stringer, Organizational benchmarking using the ISBSG Data Repository, IEEE Softw. 18 (5) (2001) 26–32.
[13] M. Mannino, S. Hong, I. Choi, Efficiency evaluation of data warehouse operations, Decis. Support Syst. 44 (4) (2008) 883–898.
[14] K. Maxwell, L. Wassenhove, S. Dutta, Software development productivity of European space, military, and industrial applications, IEEE Trans. Softw.
Eng. 22 (10) (1996) 706–718.
[15] C. Pollard, A. Cater-Steel, Justifications, strategies, and critical success factors in successful ITIL implementations in U.S. and Australian companies: an
exploratory study, Inf. Syst. Manag. 26 (2) (2009) 164–175.
[16] K. Potter, M. Smith, J. Guevara, L. Hall, E. Stegman, IT metrics: IT spending and staffing report, http://marketing.dell.com/Global/FileLib/CIO/it_metrics_it
_spending.pdf, 2010.
[17] P. Runeson, C. Wohlin, An experimental evaluation of an experience-based capture–recapture method in software code inspections, Empir. Softw. Eng.
3 (4) (1998) 381–406.
[18] M. Tsunoda, A. Monden, K. Matsumoto, A. Takahashi, T. Oshino, An empirical analysis of information technology operations cost, in: Proc. of Interna-
tional Workshop on Software Measurement, IWSM/Metrikon/Mensura, 2010, pp. 571–585.
[19] M. Tsunoda, A. Monden, K. Matsumoto, S. Ohiwa, T. Oshino, Benchmarking software maintenance based on working time, in: Proc. of Applied Comput-
ing and Information Technology, ACIT, 2015, pp. 20–27.
[20] M. Tsunoda, A. Monden, H. Yadohisa, N. Kikuchi, K. Matsumoto, Software development productivity of Japanese enterprise applications, Inf. Technol.
Manag. 10 (4) (2009) 193–205.
[21] H. Wang, H. Wang, H. Zhang, Software productivity analysis with CSBSG data set, in: International Conference on Computer Science and Software
Engineering, 2008, pp. 587–593.