sustainability

Article
Sustainable Technology Analysis Using Data
Envelopment Analysis and State Space Models
Jong-Min Kim 1 , Bainwen Sun 1 and Sunghae Jun 2, *
1 Statistics Discipline, Division of Sciences and Mathematics, University of Minnesota-Morris, Morris,
MN 56267, USA
2 Department of Big Data and Statistics, Cheongju University, Chungbuk 28503, Korea
* Correspondence: shjun@cju.ac.kr; Tel.: +82-10-7745-5677




Received: 5 May 2019; Accepted: 27 June 2019; Published: 29 June 2019


Abstract: To find sustainable technology in various areas, we propose an analytical methodology

based on data envelopment analysis (DEA) and the state space model (SSM). DEA is an analytical
method used to compare the efficiencies and performances of several items. In DEA, for sustainable
technology analysis, the items of DEA can be the technological keywords or international patent
classification (IPC) codes in patent documents. In this paper, the proposed method is used to find
the relative performance of different patent keywords using comparison and evaluation. We apply
this methodology to compare the technological efficiencies between patent keywords for sustainable
technology analysis. We apply the additive model and directional distance function of DEA to
develop the proposed methodology for building the technological structure of target technology.
In addition, we forecast the future trend of target technology using the SSM and find the area of
sustainable technology by its result. The SSM is well suited for time series forecasting on technology
analysis. We extract the IPC codes from patent documents for the SSM. In our research, we combine
the results of DEA and the SSM to find the area of technological sustainability. To illustrate the
validity and performance of our research, we conduct a case study using the patent documents used
and registered by Apple.

Keywords: data envelopment analysis; patent keywords; state space model; technological
sustainability; Apple technology

1. Introduction
Researchers have been trying to forecast future technology for various purposes [1]. For example,
using the results of technology forecasts, a company can undertake a variety of technological innovations
to improve its technological competitiveness in the market [2]. The ability to predict the future state of
technology can have a significant impact on a company. Thus, technology forecasting enables companies
to develop better research and development (R&D) strategies. It may also identify potential business
opportunities in the market and provide the opportunity to identify a new product. Recently, various
methods for technology forecasting have been researched in diverse fields using statistics and machine
learning algorithms [2–5]. Researchers have used graphical methods, ensemble modeling, Bayesian
inference, and copula modeling to analyze patent documents for technology analysis, because a patent
contains many examples of developed technology [6]. In this paper, we introduce a methodology for
technology forecasting by analyzing patent document data.
We consider additive model and directional distance function of data envelopment analysis
(DEA) for technology analysis. In previous research, DEA was used for constructing patent-enhancing
strategies in Korean industries [7]. It has been used to find the empirical measure of productive
efficiency for a decision-making unit (DMU) [8,9], and in this study, we use this method for patent data

Sustainability 2019, 11, 3597; doi:10.3390/su11133597 www.mdpi.com/journal/sustainability

Sustainability 2019, 11, 3597 2 of 19

analysis. In our research, this unit can be considered to be one of many items, such as patent keywords,
representing the sub-technology. Our research proposes a new method to address the technological
relation between patent keywords to support R&D planning in the management of technology (MOT).
In this study, we conducted two analyses of additive model and directional distance function and
combined these results to understand the sustainable technology in a given target domain. We also
considered the state space model (SSM) to forecast the future state of technology. This is similar to the
hidden Markov model (HMM), and the only difference from the HMM is the use of a continuous value
for the hidden state. The SSM is very well suited for time series forecasting [10].
In this paper, we used a local level model as the SSM for sustainable technology analysis.
This model is based on Gaussian distribution and provides good results for time series forecasting
of technology. The input data applied to the SSM are the international patent classification (IPC)
codes containing detailed information about developed technology. An IPC code is defined by
its corresponding technology [11]. Finally, we combine the results of DEA and the SSM to find
technological sustainability. Sustainable technology is a technology sustaining the technological
competitiveness of a nation or company in the global market [5,12–16]. The goal of this paper is to
find the sustainable technology structure of a target domain using DEA and the SSM. To illustrate the
validity and performance of our study, we conducted a case study using the patent documents used
and registered by Apple, because Apple is one of leading companies in technological innovation [17].
The remainder of this paper is organized as follows. First, we explain the research background
related to our paper in Section 2. In Section 3, we explain the proposed methodology of our research.
We also show the result of our case study of Apple’s patent data in Section 4. Lastly, in Section 5, we
present our conclusions and discuss our future work related to sustainable technology analysis.

2. Research Background

2.1. Data Envelopment Analysis

In 1978, Charnes, Cooper, and Rhodes introduced a method related to operations research called
data envelopment analysis [8]. DEA has been actively used by many companies and organizations
for the effective measurement and evaluation of organizational performance. The problem in DEA
is characterized by the number of decision-making units (DMUs), size of input and output, and
percentage of extreme efficient units in the data. DEA is a nonparametric method used in operations
research or economics for the performance estimation of production. DEA is also a method used to
evaluate the efficiency of a set of DMUs transforming inputs into outputs. The inputs and outputs are
represented by X = (x1 , x2 , . . . , xn ) and Y = ( y1 , y2 , . . . , ym ), respectively. A DMU produces outputs Y
from inputs X, and the efficiency is defined as follows [18]:

u1 y1 + u2 y2 + · · · + um ym
e= , (1)
v1 x1 + v2 x2 + · · · + vn xn

where V = (v1 , v2 , . . . , vn ) and U = (u1 , u2 , . . . , um ) are the weights for inputs and outputs. The V and
U are determined by linear programming called multiple form as follows [8,18]:
Xm Xm Xn Xn
e = Maxv,u ui yi,j∗ , s.t. ui yij − vi xij ≤ 0, j = 1, 2, . . . , j∗ , . . . , N, vi xi,j∗ , (2)
i=1 i=1 i=1 i=1

where vi (i = 1, n) and ui (i = 1, m) are larger than or equal to 0, respectively. In DEA, the highest
efficiency is set to 1, and the relative ratio to this value is calculated and used for analysis. If the value
of the objective function is equal to one or close to one, the evaluated DMU is efficient. It is used to
measure the productive efficiency of a DMU. In our study, DMU is a technology; it has control over
its inputs—the occurred frequencies of technological keywords. DMU can also be expected to have
control over its outputs—the technological performances of technological keywords. The technique of
DEA is based on linear programming to evaluate the efficiency of a technological keyword in target
Sustainability 2019, 11, 3597 3 of 19

Sustainability 2017, 9, x FOR PEER REVIEW 3 of 21

technology. The strategy of DEA is determined by controlling the inputs and outputs of the DMU. After
After determining the strategy, DEA is performed on the observed units using linear programming
determining the strategy, DEA is performed on the observed units using linear programming [8,9,19].
[8,9,19].
2.2. Sustainable Technology
2.2. Sustainable Technology
Most companies have built and implemented enterprise-wide strategies for sustainability and are
Most companies
committed to developing have built andtechnology.
sustainable implemented enterprise-wide
Sustainable technology strategies for sustainability
is a technology and
that can sustain
are
the committed
technological to developing sustainable
competitiveness technology.
of a company Sustainable
[2]. Leading technology
companies, suchisasa technology that can
Apple or Samsung,
sustain the technological competitiveness of a company [2]. Leading companies,
have carried out technological innovation continuously with respect to their sustainable technologies. such as Apple or
Samsung, have carried out technological innovation continuously with respect
Many studies related to sustainable technology have been performed in diverse technology domains. to their sustainable
technologies.
Jun Many Bayesian
(2018) proposed studies related to sustainable
count data modeling for technology have been sustainability
finding technological performed in [2]. diverse
This
technology domains. Jun (2018) proposed Bayesian count data modeling
applied Bayesian inference and Poisson probability model to the proposed method for extractingfor finding technological
sustainability [2]. This applied Bayesian inference and Poisson probability model to the proposed
technological relations between technologies. Park et al. (2015) developed a network model for selecting
method for extracting technological relations between technologies. Park et al. (2015) developed a
sustainable technology [5]. They used graph theory and social network analysis for their proposed
network model for selecting sustainable technology [5]. They used graph theory and social network
model, and international patent classification codes as input data. Kim et al. (2015) introduced various
analysis for their proposed model, and international patent classification codes as input data. Kim et
growth curve methods for technology management of the defense industry [13]. They also combined
al. (2015) introduced various growth curve methods for technology management of the defense
the Delphi technique with surveys and applied the combined results to the growth curve model.
industry [13]. They also combined the Delphi technique with surveys and applied the combined
In addition, sustainable technology analysis using various analytical techniques has been performed in
results to the growth curve model. In addition, sustainable technology analysis using various
diverse technology fields [4,12,14–16].
analytical techniques has been performed in diverse technology fields [4,12,14–16].
3. Sustainable Technology Analysis Using Data Envelopment Analysis and State Space Models
3. Sustainable Technology Analysis Using Data Envelopment Analysis and State Space Models
In this paper, we propose a new methodology for patent data analysis for sustainable technology
In this
analysis. paper,
First, we propose
we applied a new methodology
the additive for patentdistance
model and directional data analysis
function for of
sustainable
DEA to the technology
proposed
analysis.
methodology.First,Wewealso
applied the additive
used patent model
document dataand directional
as input data fordistance
the DEAfunction of DEApatents
model, because to the
proposed methodology. We also used patent document data as input data for
have detailed and rich information about developed technology, such as the titles, abstracts, claims, the DEA model,
because
inventors’ patents
names,have detailed
applied and rich
and issued information
dates, about
figures, etc. [1,6].developed technology,
After determining the such
targetastechnology,
the titles,
abstracts,
we claims,
collected inventors’
the patent names, applied
documents related and issued
to the targetdates, figures, from
technology etc. [1,6]. After
patent determining
databases the
all over
target technology, we collected the patent documents related to the target
the world [20,21]. Next, we extracted patent keywords from the patent documents. The keywords technology from patent
databases all over the
represent various world [20,21]. related
sub-technologies Next, wetoextracted
the targetpatent keywords
technology. In thefrom the patent
process documents.
of preprocessing
The
keyword data, we use various text mining techniques based on R data language and itsprocess
keywords represent various sub-technologies related to the target technology. In the provided of
preprocessing keyword data, we use various text mining
package [22,23]. Our text mining process is shown in Figure 1. techniques based on R data language and
its provided package [22,23]. Our text mining process is shown in Figure 1.

Figure 1. Preprocessing via the text mining technique.

Figure 1. Preprocessing via the text mining technique.
Using the preprocessing of Figure 1, we obtain a patent–keyword matrix consisting of patent
Using the preprocessing
(row) and keywords (column), ofandFigure 1, we obtain
each element of thisa matrix
patent–keyword
represents matrix consisting
the frequency of a of patent
keyword
(row)
in eachand keywords
patent (column),
document. Thisand eachiselement
matrix used asofinput
this matrix represents
data for EDA. The the frequency
keywords of apatent
and keyword are
in each
used forpatent document.
variables This matrixitems
and observational is used as input
in our model. data
Wefor EDA. The
consider keywords
the additive and patent
model of DEAare to
used for variables and observational items in our model. We consider
analyze the patent keyword data to find technological structure in the target technology. the additive model of DEA to
analyze
Thethe patentmodel
additive keyword data to outputs
maximizes find technological
and minimizes structure
inputs inatthe
thetarget
sametechnology.
time [23]. The vector is
The additive
optimized model
as follows maximizes outputs and minimizes inputs at the same time [23]. The vector
[9,19,24–28]:
is optimized as follows X[9,19,24–28]:
0 0 n Xn
Max es + e s , s. t., X𝑛j + si = Xt ,𝑖
λ j∑ λ j𝑛Y j − s0 = Y0t , si , s0 ≥ 0 ; λ j ≥ 0; j = 1, . . . , n, (3)
𝑖 0,
=1t., 𝑗=1 𝜆𝑗 𝑋𝑗 + 𝑠 = 𝑋
Max es + e’s’, js. 𝑡 ,1 ∑𝑗=1 𝜆𝑗 𝑌𝑗 − 𝑠 = 𝑌𝑡 , 𝑠 , 𝑠 ≥ 0, ; 𝜆𝑗 ≥ 0; 𝑗 =
j=
(3)
1, … , 𝑛,
Where si and s0 are the input and output slack vectors, respectively, and e and e’ are conformable
vectors. The dual problem to the above model is shown as follows [9,19,24–28]:
Sustainability 2019, 11, 3597 4 of 19

where si and s0 are the input and output slack vectors, respectively, and e and e’ are conformable
vectors. The dual problem to the above model is shown as follows [9,19,24–28]:

Sustainability 2017, 9, xMin

FOR vXt +REVIEW
PEER uYt , s. t., vX j + uY j ≥ 0; j = 1, . . . , n; v ≥ +1; u ≤ −1. 4 of(4)
21

As we maximize the input and output slacks simultaneously, the units in the optimal basis of
Min 𝑣𝑋𝑡 + 𝑢𝑌𝑡 , s. t., 𝑣𝑋𝑗 + 𝑢𝑌𝑗 ≥ 0; 𝑗 = 1, … , 𝑛; 𝑣 ≥ +1; 𝑢 ≤ −1. (4)
Equation (2) are always Pareto–Koopmans efficient unlike in the case of the oriented models [24].
UsingAs wemodel,
this maximize the the
we find input and output
efficiency slacks
of each simultaneously,
patent keyword. Wethe useunits in the
another optimal
method basis to
of DEA of
Equation
obtain the (2) are always
efficiency of thePareto–Koopmans
technology keyword. efficient unlike
This is in the distance
directional case of the oriented
function. Themodels [24].
directional
Using this
output model,
distance we findisthe
function theefficiency of each version
output-oriented patent keyword. We use
of the benefit another
function methodThe
[29–31]. of DEA
benefitto
obtain the efficiency of the technology keyword. This is directional distance function. The
function is applied to diverse optimization areas. The directional distance function is a generalized directional
output
form distance
based function
on a radial is the
model as output-oriented
follows [32,33]: version of the benefit function [29–31]. The benefit
function is applied to diverse optimization areas. The directional distance function is a generalized
form based on a radial model Max B,as s.follows + Bgx ≤ x0 , YA − Bg y ≥ y0 , A ≥ 0,
t., XA [32,33]: (5)

Max
where gx and g y are direction B, s. t.,
vectors 𝑋𝐴 +to𝐵𝑔
related ≤ 𝑥0 , 𝑌𝐴
x 𝑥(input) and−y𝐵𝑔 𝑦 ≥ 𝑦0 , and
(output) 𝐴 ≥ B0,is un-efficiency measure.
(5)
This
where is formulated
𝑔𝑥 and 𝑔𝑦on undesirable
are outputsrelated
direction vectors as followsto x[32,33]:
(input) and y (output) and B is un-efficiency
measure. This is formulated on undesirable outputs as follows [32,33]:
Max B, s. t., XA + Bgx ≤ x0 , YA − Bg y ≥ y0 , B0 A + Bgb ≤ b0 , A ≥ 0. (6)
Max B, s. t., 𝑋𝐴 + 𝐵𝑔𝑥 ≤ 𝑥0 , 𝑌𝐴 − 𝐵𝑔𝑦 ≥ 𝑦0 , 𝐵′ 𝐴 + 𝐵𝑔𝑏 ≤ 𝑏0 , 𝐴 ≥ 0. (6)
Equation (4) shows a directional between good (y) and bad (b) outputs. Using a direction vector
Equation (4) shows a directional between good (y) and bad (b) outputs. Using a direction vector
gb , we control the bad outputs (b). In addition, all direction vectors are non-negative. We control the
𝑔𝑏 , we control the bad outputs (b). In addition, all direction vectors are non-negative. We control the
good and bad outputs at the same time. Both have meaning in technology analysis. In the process
good and bad outputs at the same time. Both have meaning in technology analysis. In the process of
of finding the final sustainable technology area, good output strengthens the technology association,
finding the final sustainable technology area, good output strengthens the technology association,
and bad output weaken the association. In our study, we combine the results of the additive model
and bad output weaken the association. In our study, we combine the results of the additive model
and directional distance function to find the technological sustainability in the target technology. The
and directional distance function to find the technological sustainability in the target technology. The
technological structure between two keywords X and Y is shown in Figure 2.
technological structure between two keywords X and Y is shown in Figure 2.

Figure 2. Technological structure between two keywords.

Figure 2. Technological structure between two keywords.
In Figure 2, the input keyword X affects the output keyword Y by two results of the additive
In Figure 2, the input keyword X affects the output keyword Y by two results of the additive
model and directional distance function. The solid line shows a meaningful efficiency in both models,
model and directional distance function. The solid line shows a meaningful efficiency in both models,
and the dotted line shows a case in which only one model has significant efficiency. That is, the solid
and the dotted line shows a case in which only one model has significant efficiency. That is, the solid
line shows a greater influence. We can build this structure between all the keywords related to the
line shows a greater influence. We can build this structure between all the keywords related to the
target technology. The following procedure summarizes the proposed method described above.
target technology. The following procedure summarizes the proposed method described above.
Step 1: Determining the target technology;
Step 1: Determining the target technology;
Step 2: Collecting patent documents related to the target technology;
Step 2: Collecting patent documents related to the target technology;
Step 3: Preprocessing collected patent documents using text mining techniques;
Step 3: Preprocessing collected patent documents using text mining techniques;
Step 4: Extracting technological keywords for constructing the patent–keyword matrix;
Step 4: Extracting technological keywords for constructing the patent–keyword matrix;
Step 5: Performing additive model using the patent–keyword matrix;
Step 5: Performing additive model using the patent–keyword matrix;
Step 6: Performing directional distance function using the patent–keyword matrix;
Step 6: Performing directional distance function using the patent–keyword matrix;
Step 7: Combining the results of Step 5 and Step 6 to find the technological structure
Step 7: Combining the results of Step 5 and Step 6 to find the technological structure
Step 8: Building R&D strategy for the technological sustainability of the target technology.
Step 8: Building R&D strategy for the technological sustainability of the target technology.
In next case study, we will determine five variables (technological keywords) that are used for
In next case study, we will determine five variables (technological keywords) that are used for
variables X and Y. Holding variable Y as constant, we change variable X through the other four
variables X and Y. Holding variable Y as constant, we change variable X through the other four
variables and observe how they interact with variable Y, respectively. Then, we obtain the results of
variables and observe how they interact with variable Y, respectively. Then, we obtain the results of
each interaction and visualized them. We search the patent documents used and registered by Apple.
Using text mining techniques, we also extract keywords and built structured data. This data type is
a matrix of patents (rows) and keywords (columns). The element of this matrix is the frequency value
of a specific keyword in the patent document. We use this matrix for input and output measurements
of DEA.
Sustainability 2019, 11, 3597 5 of 19

each interaction and visualized them. We search the patent documents used and registered by Apple.
Using text mining techniques, we also extract keywords and built structured data. This data type is a
matrix of patents (rows) and keywords (columns). The element of this matrix is the frequency value of
a specific keyword in the patent document. We use this matrix for input and output measurements
of DEA.
In this paper, we evaluate the state space model as another approach to patent analysis for
sustainable technology. The SSM is similar to the hidden Markov model (HMM), and the only
difference is that the hidden state of the HMM is discrete and the state of the SSM is continuous. The
general form of the SSM is as follows [10]:

Zt = G(µt , Zt−1 , εt ), Yt = H(Zt , µt , δt ), (7)

where Zt , µt , and Yt are the hidden state, input, and observation, respectively. G and H are the
transition and observation models. εt and δt are the  noises of system and observation at time t. The
goal of the SSM is to estimate the belief state P(Zt Y1:t ) and predict future observation P(Yt+1 Y1:t )
using the beliefs of the hidden state. To understand the technological trend for sustainable technology,
we use the SSM for time series forecasting, because the SSM is an efficient approach to time series
forecasting [10]. The SSM makes a generative model using hidden variable integration and posterior
prediction. This is local level model based on Gaussian distribution as follows [10]:

Yt = Zt + εY Z
t , Zt = Zt−1 + εt , (8)

where Yt and Zt are already explained in Equation (5) and εY t and εt are distributed to Gaussian with
Z

same mean 0 and variances VY and VZ , respectively. The forecasting process of the local level model is
carried out as follows:
{Zt−1 } → {Zt } → {Yt }. (9)

In our research, we develop a structural time series model (local level model) to find the
technological trend. In this paper, we use the international patent classification codes by year in the
time series model, because the IPC codes contain various technological descriptions on the target
domain. We combine the results of DEA and the SSM to find technological areas for technological
sustainability of a company. Thus, the final goal of our research is to build the R&D strategy for
the technological sustainability of the target technology. A company can improve its technological
competitiveness in the market using the results of our research. Next, we illustrate a case study
implementing our proposed method. In this study, we show the technological efficiencies of five
patent keywords (system, data, device, media, user) using DEA. We also illustrate the technological
structure of high-ranked IPC codes using the SSM. Combining the results of DEA and the SSM, we
find sustainable areas of the given technology.

4. Case Study
To show the performance and validity of our research, we carried out a case study on Apple
technology. We retrieved patent documents used by Apple [20,21]. The documents used in this study
were filed between 1980 and 2010. We used R and its Package to compute the efficiencies by using
additive model and directional distance function [22,23,34]. First, we extracted the technological
keywords from the retrieved patent documents via text mining techniques [22]. Based on expert
opinions and previous studies [3,35], we identified five key keywords as follows: system, media,
data, user, and device. They represent Apple’s sub-technologies. Using them, we performed five
experiments using DEA. For each experiment, we changed the combination of keywords used as input
and output variables. We did so to understand the association between Apple’s sub-technologies
and the keywords. In this paper, we conclude that these five technology keywords are the basis of
the technology that supports Apple’s innovative products. In addition, the keywords were used
Sustainability 2019, 11, 3597 6 of 19

as variables in our models. Each keyword represents a sub-technology of Apple. For example, the
data keyword represents the Apple technology related to data collecting, integrating, preprocessing,
analyzing, visualizing, etc. We used a total of 8114 patent documents associated with DMUs. We
applied additive model and directional distance function to measure productivity for the efficiency of
Apple technological keywords. Table 1 shows the results of an additive model in which the Y variable
is system and the others are X variables.

Table 1. Result of additive model: Y = system, X = (media, data, user, device)

Year Media Data User Device

1980 167.00 1.26 0.00 0.00
1981 169.00 3.26 2.00 19.57
1982 167.00 0.00 6.00 17.57
1983 167.00 6.30 0.00 0.00
1984 169.00 2.00 8.00 2.00
1985 164.00 14.65 9.00 48.00
1986 169.00 18.39 2.00 2.00
1987 158.00 6.13 3.00 129.58
1988 161.00 4.08 0.00 29.14
1989 155.00 52.29 0.00 54.00
1990 348.00 22.03 21.04 0.00
1991 410.40 52.41 30.49 49.00
1992 315.00 94.90 88.13 62.00
1993 0.00 197.25 245.35 51.40
1994 141.00 63.03 134.00 6.10
1995 0.00 74.28 50.96 0.00
1996 0.00 0.00 0.00 46.00
1997 265.00 74.90 171.76 45.28
1998 338.00 0.13 126.22 0.00
1999 376.00 4.45 227.71 63.73
2000 341.00 23.40 341.00 92.20
2001 310.00 0.00 279.00 63.25
2002 379.00 107.30 373.00 260.23
2003 309.00 49.45 288.00 127.48
2004 391.00 79.82 290.00 258.00
2005 647.00 128.90 355.00 413.00
2006 587.00 96.63 445.77 343.00
2007 242.10 43.61 185.20 70.48
2008 0.00 0.00 0.00 0.00
2009 505.00 255.00 351.00 909.00
2010 0.00 252.27 108.25 435.58

We maintained variable Y (system) as a constant and changed variables Xs (data, device, media,
user). As the results shown in Table 1, all the Xs has a technological impact on system. However,
during the first 10 years, only media had an impact on the system, and its effect was mainly on its
stability. More precise implications occurred after 1990, except for device. The variable of data had two
peaks and its trend showed growth peak and then decline with 2010 being the most substantial time.
With respect to the variable device, the apparent impact occurred around 2000, and its roughest time
occurred approximately in 2009 then decreased in approximately 2010. For the media variable, a peak
occurred around 1992 and then reduced to zero. After 1995, it increased steadily and tended to stabilize.
In 2005, it reached its highest essential time and then decreased. For the user variable, the peak occurred
around 1993, and after 1996, the growth tended to stabilize and then reduce. Figure 3 illustrates the
results of the directional distance function for Y = system and X = (media, data, user, device).
Sustainability 2017, 9, x FOR PEER REVIEW 7 of 21
Sustainability 2017, 9, x FOR PEER REVIEW 7 of 21
Sustainability 2019, 11, 3597 7 of 19

Figure 3. Result of directional distance function: Y = system, X = (media, data, user, device).
Figure 3. Result of directional distance function: Y = system, X = (media, data, user, device).
Figure 3. Result of directional distance function: Y = system, X = (media, data, user, device).
According to the results of the directional distance function, during the first 10 years, there was
According
no impact for allto thevariables.
results of the directional distance function, duringtothe
thefirst 10 years, there was
According tothe
the results ofAfter 1990, the growth
the directional trend
distance was similar
function, during the additive
first model,
10 years, except
there was
no impact
forimpact for
the media all the variables.
variable. After
The media 1990, the growth trend was similar to the additive model, except
no for all the variables. After variable
1990, thehad no obvious
growth trend was impacts
similarbefore 2000, butmodel,
to the additive after 2000, it
except
for the media
increased variable.
rapidly and The media
arrived at its variable
peak in had before
2005 no obvious impacts
decreasing. before
Using the2000, butofafter
results the 2000, it
additive
for the media variable. The media variable had no obvious impacts before 2000, but after 2000, it
increased
model andrapidly and arrived
directional distanceatfunction,
its peak in 2005 before
a diagram decreasing.
of the Usingsustainability
technological the results of the additive
of system is
increased rapidly and arrived at its peak in 2005 before decreasing. Using the results of the additive
model and
shown and directional
in Figure 4. distance function, a diagram of the technological sustainability of system is
model directional distance function, a diagram of the technological sustainability of system is
shown in Figure 4.
shown in Figure 4.

Figure 4. Diagram of the technological sustainability of system.

Figure 4. Diagram of the technological sustainability of system.
Figure 4. Diagram of the technological sustainability of system.
The solid line shows efficiency in both the additive model and directional distance function,
The
and the solid line
lineshows efficiency in in
both theofadditive
the twomodel and directional distance function, was
and
Thedotted
solid line shows
shows efficiency
efficiency in bothone
the additive results.
model and Therefore,
directional system technology
distance function, and
the dotted
continuouslyline shows
influenced efficiency
by the mediain one of the
technology, two results. Therefore, system technology was
the dotted line shows efficiency in one of the and
two itresults.
can be seen that it was
Therefore, affected
system by user was
technology and
continuously
data influenced
technologies byNext,
recently. the media
the technology,
results of the and it can
additive be seen
model in that itthe
which wasY affected is
variable bythe
user and
media
continuously influenced by the media technology, and it can be seen that it was affected by user and
and the X variables are system, data, user, and device are shown in Table 2.
Sustainability 2019, 11, 3597 8 of 19

Table 2. Result of additive model: Y = media, X = (system, data, user, device).

Year System Data User Device

1980 3.88 1.56 0.00 0.00
1981 0.00 1.56 0.00 4.73
1982 3.88 0.00 1.78 4.73
1983 3.88 7.82 0.00 0.00
1984 0.00 0.00 1.78 0.00
1985 9.69 21.90 3.56 14.19
1986 0.00 20.34 0.00 0.00
1987 21.32 18.77 3.56 42.27
1988 15.50 12.51 1.78 9.46
1989 27.13 79.78 3.56 18.92
1990 77.34 72.21 17.15 5.04
1991 46.20 88.55 11.37 12.08
1992 141.29 203.61 56.32 35.00
1993 327.53 447.00 160.24 81.72
1994 506.00 437.00 175.73 114.53
1995 647.00 590.00 328.07 181.69
1996 633.00 489.00 158.59 221.05
1997 346.35 305.81 120.88 70.08
1998 226.45 127.94 60.49 0.00
1999 135.92 81.81 96.29 21.82
2000 161.61 127.53 158.59 54.36
2001 267.58 159.15 143.93 52.17
2002 190.14 221.16 235.27 85.06
2003 338.00 245.17 237.44 35.34
2004 256.00 204.49 170.71 69.40
2005 0.00 0.00 0.00 0.00
2006 32.68 0.00 148.09 0.00
2007 144.25 138.99 178.10 320.74
2008 0.00 0.00 0.00 0.00
2009 133.96 247.12 132.44 648.00
2010 411.00 609.00 362.00 955.00

In Table 2, the growth trends of all variables are similar. There was no impact before 1990, and the
impact increased rapidly after 1990 until it peaked in 1995. Then it decreased around 2000. There was
increased fluctuation
Sustainability 2017, 9, x after 2000.
FOR PEER This was because the rates of fluctuation are different. Figure
REVIEW 9 of521shows
the results of the directional distance function for media (Y variable).

Figure 5. Result of directional distance function: Y = media, X = (system, data, user, device).
Figure 5. Result of directional distance function: Y = media, X = (system, data, user, device).

According to the results of the directional distance function in Figure 5, the main growth trends
were similar to the additive model; the scale of efficiency was merely less than that in additive model.
From the results of Table 2 and Figure 5, we developed a technological diagram for the model in
which media was the Y variable and others were the X variables (system, media, user, device) in
Figure 6.
Sustainability 2019, 11, 3597 9 of 19

According to the results of the directional distance function in Figure 5, the main growth trends
were similar to the additive model; the scale of efficiency was merely less than that in additive model.
Figure 5. Result of directional distance function: Y = media, X = (system, data, user, device).
From the results of Table 2 and Figure 5, we developed a technological diagram for the model in which
media was the Y variable
According and others
to the results of the were the Xdistance
directional variables (system,
function media,
in Figure user,
5, the device)
main growthin trends
Figure 6.
We
were similar to the additive model; the scale of efficiency was merely less than that in additive model. and
found that the technology of media was affected by system, data, and user strongly
From the
influenced by results
deviceof Table 2 and
technology Figure 5,
weakly. we developed
Next, a technological
Table 3 provides diagram
the result for theand
of Y (data) model in
X (system,
which
media, user,media wasusing
device) the Ythe
variable andmodel.
additive others were the X variables (system, media, user, device) in
Figure 6.

Figure 6. Diagram of the technological sustainability of media.

Figure 6. Diagram of the technological sustainability of media.
Table 3. Result of additive model: Y = data, X = (system, media, user, device).
We found that the technology of media was affected by system, data, and user strongly and
influenced Year
by device technology weakly. Next, Table
System Media 3 provides theUser
result of Y (data)Device
and X (system,
media, user, device) using the17.43
1980 additive model. 288.00 12.00 12.00
1981 12.00 288.00 12.00 27.86
1982 18.43 289.00 32.00 28.86
1983 13.43 284.00 8.00 8.00
1984 13.00 289.00 32.00 13.00
1985 12.57 275.00 37.00 45.37
1986 0.00 276.00 0.00 0.00
1987 30.86 277.00 39.00 131.73
1988 26.71 281.00 24.00 36.71
1989 0.00 238.00 0.00 22.43
1990 45.53 316.50 35.69 0.00
1991 11.03 355.17 19.54 19.80
1992 12.20 232.50 21.08 5.04
1993 0.00 0.00 39.54 0.00
1994 181.35 153.05 60.06 60.41
1995 216.37 0.00 0.02 0.00
1996 296.99 41.42 0.00 133.00
1997 107.38 243.32 83.69 90.81
1998 133.40 366.57 116.46 31.04
1999 88.41 388.60 189.00 86.10
2000 77.09 293.42 252.00 128.02
2001 143.90 340.81 222.25 160.63
2002 52.08 326.06 236.20 217.06
2003 161.21 326.06 221.00 160.44
2004 145.24 392.31 203.39 248.00
2005 85.15 609.00 223.52 356.00
2006 196.69 589.00 394.00 309.17
2007 209.33 448.00 284.00 232.50
2008 190.00 536.00 219.00 349.42
2009 36.95 356.00 123.49 490.38
2010 0.00 0.00 0.00 0.00

In Table 3, there was no impact in the first 10 years except the media variable. The media variable
had a stability effect in the first 10 years. Then, around 1990, it arrived peak and decreased to zero.
Sustainability 2019, 11, 3597 10 of 19

In the following years, it increased steadily and declined after 2005. For the system variable, its peak
was reached in around 1995. Then it reduced but also steadily grew over the next few years. With
respect to the device variable, it increased around 1995, showing slow but steady growth. The user
variable steadily increased around 1995 and remain unchanged for around five years. It was peaked in
Sustainability
2005 and then 2017,decreased
9, x FOR PEER
toREVIEW 11 of 21
zero. Another result of the data keyword (variable Y) by the directional
distance function is shown in Figure 7.

Sustainability 2017, 9, x FOR PEER REVIEW 11 of 21

Figure 7. Result of the directional distance function: Y = data, X = (system, media, user, device).
Figure 7. Result of the directional distance function: Y = data, X = (system, media, user, device).
Figure 7. Result of the directional distance function: Y = data, X = (system, media, user, device).
According to the results of the directional distance function in Figure 7, the main impact was
According
similar that oftothe
toAccording the results
toadditive
the of the
model
results of thedirectional
except withdistance
directional function
respectfunction
distance to inin
the media Figure 7, the
variable.
Figure 7, the Themain
main impact
wasofwas
efficiency
impact the
similar
media to that
began atof the
zero additive
and stayedmodel
in except
this with
situation respect
for around to the
20 media
years. variable.
After 2000, The
it
similar to that of the additive model except with respect to the media variable. The efficiency of the efficiency
increased of the
rapidly
media began
media
and peaked at2005,
began
in zero and
at zero stayed
and
before stayedininthis
decreasing thissituation
tosituation for around
for around
zero. Using 2020years.
years.
the results After
ofAfter
Table 32000,
2000, it increased
it increased
and Figure werapidly
5,rapidly
built a
and peaked
and peaked
diagram in 2005,
of the before
in 2005, decreasing
before
sustainable decreasing
technology totodata,
of zero. Using
zero.as
Using
shown thein
the results
results ofof
Figure 8.Table
Table 3 and
3 and Figure
Figure 5, we5,built
we built
a a
diagram
diagram of sustainable
of the the sustainable technologyofofdata,
technology data, as
as shown
shown in inFigure
Figure8. 8.

Figure Diagram
8. 8.
Figure Diagramof
ofthe
the technological sustainability
technological sustainability of data.
of data.

Based on Figure 6, we
Figure knew thatofdata
8. Diagram technology was
the technological most affected
sustainability by media technology.
of data.
Moreover, media technology had the least impact on data technology. Thus, the system and user
have influenced
Based on Figure data technology
6, we continuously
knew that since 1990.was
data technology Table 4 represents
most affected the
by additive model
media technology.
results media
Moreover, with respect to user technology.
technology had the least impact on data technology. Thus, the system and user
have influenced data technology continuously since 1990. Table 4 represents the additive model
Sustainability 2019, 11, 3597 11 of 19

Based on Figure 6, we knew that data technology was most affected by media technology.
Moreover, media technology had the least impact on data technology. Thus, the system and user have
influenced data technology continuously since 1990. Table 4 represents the additive model results with
respect to user technology.

Table 4. Result of additive model: Y = user, X = (system, media, data, device).

Year System Media Data Device

1980 3.57 90.00 2.25 1.00
1981 1.00 90.00 2.25 7.83
1982 2.57 89.00 0.00 6.83
1983 3.57 90.00 7.24 1.00
1984 0.00 89.00 0.00 0.00
1985 5.44 88.00 16.46 19.48
1986 1.00 90.00 17.21 1.00
1987 13.16 88.00 13.97 60.45
1988 10.30 89.00 9.98 13.66
1989 17.02 88.00 62.61 26.31
1990 43.34 127.50 50.36 3.93
1991 26.32 162.17 67.58 20.86
1992 63.81 105.50 133.13 25.17
1993 128.51 0.00 271.45 25.86
1994 290.11 86.00 259.19 61.29
1995 383.00 0.00 363.77 39.07
1996 458.84 55.92 329.57 164.15
1997 164.96 119.00 181.18 45.00
1998 121.61 163.00 78.70 3.17
1999 39.54 135.00 17.57 0.00
2000 19.97 55.92 15.26 0.00
2001 100.63 110.00 53.42 17.00
2002 4.04 85.00 65.71 32.03
2003 119.75 99.68 92.35 0.00
2004 122.06 179.24 120.25 86.28
2005 45.87 369.55 104.23 176.60
2006 0.00 119.83 0.00 0.00
2007 26.35 0.00 33.31 1.31
2008 0.00 0.00 0.00 0.00
2009 103.54 226.74 262.00 677.00
2010 156.00 0.00 394.00 760.00

According to the additive model result in Table 4, comparing data with the system, the primary
growth trends were similar. There was no efficiency in the first 10 years, but after 1990, it increased and
peaked in 1995, before decreasing to zero. The main difference was that the data variable increased
stability, but the system variable showed increased fluctuation from 2000 to 2005; after 2005, the data
variable multiplied, but the system variable increased slowly. The device variable appeared to show
inefficiency before 2010 before demonstrating a sudden increase in influence in 2010. For the media
variable, the first 10 years were efficient, and the growth rate fluctuated until 2010; the peak occurred
around 2005. In addition, we visualized the result of the directional distance function for the user
variable in Figure 9.
According to the results of the directional distance function in Figure 9, the main trends were
similar to the results of the additive model except for the media variable. There was no significant
impact before 2000. After that, the fluctuation increased until 2010, and the peak occurred around 2005.
Figure 10 shows the technological diagram for the sustainability of user technology.
Sustainability 2017, 9, x FOR PEER REVIEW 13 of 21
Sustainability 2019, 11, 3597 12 of 19

Figure 9. Result of the directional distance function: Y = user, X = (system, media, data, device).

According to the results of the directional distance function in Figure 9, the main trends were
similar to the results of the additive model except for the media variable. There was no significant
impact before 2000. After that, the fluctuation increased until 2010, and the peak occurred around
Figure10
2005. Figure Result of
9. shows thethe directional distance
technological diagramfunction: = user, X = (system,
for theY sustainability media,
of user data, device).
technology.
Figure 9. Result of the directional distance function: Y = user, X = (system, media, data, device).

According to the results of the directional distance function in Figure 9, the main trends were
similar to the results of the additive model except for the media variable. There was no significant
impact before 2000. After that, the fluctuation increased until 2010, and the peak occurred around
2005. Figure 10 shows the technological diagram for the sustainability of user technology.

Figure 10. Diagram of the technological sustainability of user.

Figure 10. Diagram of the technological sustainability of user.
In Figure 10, it can be observed that the technology of the media affected the technology of the user
In Figure 10, it can be observed that the technology of the media affected the technology of the
continuously. Since 1990, system and data technologies have had strong impacts on user technology.
user continuously. Since 1990, system and data technologies have had strong impacts on user
On the other hand, it can be seen that device technology had a weak influence on user technology after
technology. On the other hand, it can be seen that device technology had a weak influence on user
2000. Lastly, we show the results of the additive model for device (Y variable) in Table 5.
technology after 2000. Lastly, we show the results of the additive model for device (Y variable) in
In the results of Table 5, it10.isDiagram
Figure apparentofthat
the all the variables
technological were inefficient
sustainability of user.during the first 10 years.
Table 5.
Comparing data with the system, the overall increasing trend was same, it increased after 1990, and
the peak arrived
In Figure 10,around
it can be1995 and then
observed thatdecreased until 2000.
the technology of theAfter 2000,
media it grew
affected theslowly but steadily.
technology of the
The singular
user difference
continuously. Sincewas thatsystem
1990, the scale of efficiency
and for the system
data technologies have washad higher
strong than that on
impacts for data.
user
Comparing media
technology. On thewithotheruser,
hand, theitgrowth
can betrend before
seen that 2000 technology
device was similar.hadHowever,
a weakafter 2000, media
influence on useris
increased rapidly
technology and peaked
after 2000. Lastly, at we2005
showbefore decreasing
the results to additive
of the zero. Themodel
user also
for changed
device (Yafter 2000 but
variable) in
insignificantly;
Table 5. the maximum was reached around 2005, and then it quickly reduced. The result of the
directional distance function for device is represented in Figure 11.
Sustainability 2019, 11, 3597 13 of 19

Table 5. Result of additive model: Y = device, X = (system, media, data, user).

Year System Media Data User

1980 7.93 58.00 5.00 3.00
1981 0.00 55.00 2.00 0.00
1982 4.93 55.00 0.00 12.00
1983 7.93 58.00 13.00 3.00
1984 3.00 58.00 3.00 15.00
1985 6.32 49.00 21.55 18.00
1986 3.00 58.00 28.78 3.00
1987 3.11 31.00 0.00 0.00
1988 16.72 52.00 13.00 9.00
1989 25.51 46.00 84.26 15.00
1990 101.99 149.57 87.48 45.61
1991 60.48 167.72 103.57 36.28
1992 164.33 130.57 217.66 77.82
1993 361.57 0.00 460.92 173.85
1994 665.72 96.44 430.25 175.10
1995 917.67 41.54 648.18 320.97
1996 866.83 0.00 432.94 65.84
1997 403.40 141.98 310.30 140.92
1998 317.15 245.85 171.49 129.03
1999 176.12 190.52 95.67 149.31
2000 175.45 131.00 116.42 196.84
2001 321.66 175.15 161.02 189.82
2002 226.48 276.58 226.54 314.69
2003 500.22 394.15 305.80 375.90
2004 434.20 573.16 266.73 312.48
2005 222.07 945.00 164.94 257.00
2006 572.63 955.00 329.17 760.00
2007 373.49 612.00 242.77 448.00
2008 0.00 0.00 0.00 0.00
Sustainability 2017,2009
9, x FOR PEER REVIEW
0.00 307.00 11.15 0.00 15 of 21
2010 0.00 0.00 168.00 0.00

Figure 11. Result of the directional distance function: Y = device, X = (system, media, data, user).
Figure 11. Result of the directional distance function: Y = device, X = (system, media, data, user).
According to the results of the directional distance function in Figure 11, the main growth trends
According to the
were similar to the results
additive of thefor
model directional distanceSo,
all the variables. function in Figurea11,
we constructed the main
diagram growth
of the trends
sustainable
were similar
technology fortothethe additive(device)
Y variable model andfor Xallvariables
the variables. So,media,
(system, we constructed a Figure
data, user) in diagram 12. of the
sustainable technology for the Y variable (device) and X variables (system, media, data, user) in
Figure 12.
 Figure 11. Result of the directional distance function: Y = device, X = (system, media, data, user).

According to the results of the directional distance function in Figure 11, the main growth trends
were similar to the additive model for all the variables. So, we constructed a diagram of the
Sustainability 2019, 11, 3597 14 of 19
sustainable technology for the Y variable (device) and X variables (system, media, data, user) in
Figure 12.

Figure 12. Diagram of the technological sustainability of device.

Figure 12. Diagram of the technological sustainability of device.
In Figure 12, the technology of device was influenced by the technologies of system, data, and
In Figure 12, the technology of device was influenced by the technologies of system, data, and
user continuously since 1990. Using all the results of the additive model and the directional distance
user continuously since 1990. Using all the results of the additive model and the directional distance
function, we show the efficiency ranking of the Y and X variables in Table 6.
function, we show the efficiency ranking of the Y and X variables in Table 6.
Table 6. Efficiency ranking of the Y and X variables.

Y X
system media (1st), data (2nd), user (2nd), device (4th)
media system (1st), data (1st), user (1st), device (4th)
data media (1st), system (2nd), user (3rd), device (4th)
user media (1st), system (2nd), data (2nd), device (4th)
device system (1st), data (1st), user (1st), media (4th)

In this paper, we determined that Apple’s technology consists of five sub-technologies (system,
media, data, user, and device), and we computed the mutual efficiency of the five technologies.
In Table 6, we found that the media, system, and data technologies are important for the technological
sustainability of Apple. Next, we performed the SSM using R and its packages related to SSM [23,36–39].
We selected top four IPC codes from Apple’s patent documents.
We identified the technological definition of each IPC code in the World Intellectual Property
Organization (WIPO) [11]. Table 7 represents the major technologies of Apple by IPC code. Thus,
we used these four IPC codes for the SSM. Figure 13 shows the technological trends of the observed,
smoothed, and forecasted levels, as well as 50% probability limits.

Table 7. Four international patent classification (IPC) codes of Apple’s patents.

IPC Codes Technological Descriptions

G06F Digital data processing based on computational methods
G09G Device controlling using static means
H04N Pictorial communication
G06K Recognition and presentation of data

The forecasted levels of G06F and G09G were smaller than the observed levels. In addition, the
forecasted level of H04N did not reached the observed level. In comparison, the forecasted level of
G06K tended to increase compared to the observed level. From the result show in Figure 13, we know
 smoothed, and forecasted levels, as well as 50% probability limits.

Table 7. Four international patent classification (IPC) codes of Apple’s patents.

IPC codes Technological descriptions

Sustainability 2019, 11,
G06F3597 Digital data processing based on computational methods 15 of 19

G09G Device controlling using static means

H04N Pictorial
that Apple should keep sustaining technology based communication
on G06K to increase technological competitiveness.
G06K Recognition and
We also show the prediction errors of the four IPC codes in presentation
Figure 14. of data

Sustainability 2017, 9, x FOR PEER REVIEW 17 of 21

The forecasted levels of G06F and G09G were smaller than the observed levels. In addition, the
forecasted level of H04N did not reached the observed level. In comparison, the forecasted level of
G06K tended to increase compared to the observed level. From the result show in Figure 13, we know
that Apple should keep sustaining technology based on G06K to increase technological
competitiveness. We also show the prediction errors of the four IPC codes in Figure 14.
Figure 13. Forecasts and probability limits of the four IPC codes.

Figure 13. Forecasts and probability limits of the four IPC codes.

Figure 14. Prediction errors of the four IPC codes.

Figure 14. Prediction errors of the four IPC codes.

We know that prediction errors have been getting bigger in recent years than in the past. A large
error means that the risk is high. However, from a sustainability point of view, it is necessary to
constantly manage technological areas with a large error (risk). In the results of Figure 14, we found
that the technological risks of H04N and G06K were larger than those of the other IPC codes. So, we
 Sustainability 2019, 11, 3597 16 of 19

We know that prediction errors have been getting bigger in recent years than in the past. A large
error means that the risk is high. However, from a sustainability point of view, it is necessary to
constantly manage technological areas with a large error (risk). In the results of Figure 14, we found
that the technological risks of H04N and G06K were larger than those of the other IPC codes. So, we
should manage technologies based on H04N and G06K. Lastly, we present the sustainability forecasts
Sustainability
for the four2017,IPC
9, x FOR PEER
codes REVIEW
using the forecasts and probability limits via the SSM in Figure 15. 18 of 21the
From
results of Figures
Sustainability 2017, 9, x 13–15, weREVIEW
FOR PEER found that technologies based on H04N and G06K are important18 forofthe
21
sustainability of Apple’s technologies.

Figure 15. Sustainability forecasts for the four IPC codes.

Figure 15. Sustainability forecasts for the four IPC codes.

From the result of Figure

Figure15,
15. we can see the
Sustainability growthfor
forecasts potential
the four of
IPCthe H04N and G06K codes. This
codes.
From the result of Figure 15, we can see the growth potential of the H04N and G06K codes. This
is because their forecast trends show an upward trend until 2020. To understand Apple’s technological
is because their forecast trends show an upward trend until 2020. To understand Apple's
From the result
sustainability, of Figure
we must 15, we can
simultaneously see the the
consider growth potential
results of DEAofandthe the
H04N and
SSM. G06K16codes.
Figure showsThis
the
technological sustainability, we must simultaneously consider the results of DEA and the SSM. Figure
is becauseresults
combined their of
forecast
the twotrends show an upward trend until 2020. To understand Apple's
approaches.
16 shows the combined results of the two approaches.
technological sustainability, we must simultaneously consider the results of DEA and the SSM. Figure
16 shows the combined results of the two approaches.

Figure 16. Technological sustainability of Apple.

Figure 16. Technological sustainability of Apple.

Figure 16. Technological sustainability of Apple.

In Figure 16, the DEA results provide the technological relationship between patent keywords
of Apple. We considered only the first relation between technological keywords from the efficiency
In Figure 16, the DEA results provide the technological relationship between patent keywords
ranking of the DEA results. It can be seen that the media keyword has a technological relation with
of Apple. We considered only the first relation between technological keywords from the efficiency
the user, data, and system keywords. In addition, the system and device keywords are connected to
ranking of the DEA results. It can be seen that the media keyword has a technological relation with
each other. From the DEA perspective, we have confirmed that media and system technologies are
Sustainability 2019, 11, 3597 17 of 19

In Figure 16, the DEA results provide the technological relationship between patent keywords
of Apple. We considered only the first relation between technological keywords from the efficiency
ranking of the DEA results. It can be seen that the media keyword has a technological relation with
the user, data, and system keywords. In addition, the system and device keywords are connected
to each other. From the DEA perspective, we have confirmed that media and system technologies
are important factors in Apple’s technological sustainability. From the perspective of the SSM, we
found that two IPC codes, H04N and G06K, are important for Apple’s sustainable technology. The IPC
codes of H04N and G06F represent the technologies of ‘pictorial communication’ and ‘recognition and
presentation of data’, respectively. Therefore, Apple needs to continue to research and develop ‘image
and data technologies based on media and system technologies’ for its technological sustainability.

5. Conclusions
In this study, we proposed a methodology consisting of two models, DEA and the SSM, to
analyze the technological relationship of the target technology. We used patent keywords and IPC
codes extracted from patent documents for the proposed methodology. The target domain of our
case study is Apple technology, because Apple is one of the leading companies in the technological
innovation markets. We considered Apple’s top five keywords (system, data, user, device, media)
and four IPC codes (G06F, G09G, H04N, G06K) in our case study. Of course, the list of keywords
used by the analyst can be changed, but the method of constructing the technological structure
between given keywords can follow the proposed steps in this paper. Using the results of DEA,
we found technological relationships between sub-technologies (patent keywords). In addition, we
forecasted the future trends of sub-technologies (IPC codes) via the result of the SSM. In the DEA
result, we could see that technologies based on ‘media’ and ‘system’ are connected to each other
and others. This means that Apple needs to research and develop these areas for the sustainability
of Apple’s technology. Furthermore, via the SSM result, we confirmed that technologies based on
H04N (pictorial communication) and G06K (recognition and presentation of data) are important to
Apple’s technological sustainability. Finally, by combining the results of the two analyses, we were
able to identify ‘Image and data technologies based on media and system technologies’ as Apple’s
sustainable technology.
In this paper, we extended the usage of traditional DEA to sustainable technology analysis. We
used technological keywords extracted from patent documents as DEA input and output variables.
This is a new attempt, different from existing DEA. For a technological keyword that becomes the
output variable, all other keywords are input variables. The relative efficiency of the input keywords
with respect to the technological keyword corresponding to the output variable was evaluated, and
the sub-technologies for constructing the sustainable technology were defined by searching for the
keywords corresponding to the high-ranked input keywords. The frequency values of the technological
keywords were used to measure the consumption and gain for DEA at the same time. In addition,
in this paper, sustainable technologies were presented comprehensively, considering both the results
of the SSM, as well as the DEA. In our future research, we plan to apply Bayesian inference to the
traditional SSM to improve the predictive performance of the general SSM and to consider non-Gaussian
distributions for more sophisticated modeling. We will also study more advanced and hybrid models
using statistics and machine learning algorithms, such as Bayesian inference with Markov chain Monte
Carlo (MCMC) and deep learning with convolutional neural network (CNN).

Author Contributions: J.-M.K. and B.S. designed this research and collected the data set for the experiment. S.J.
analyzed the data, wrote the paper, and performed all the research steps. All the authors cooperated to revise
the paper.
Funding: This research was supported by Basic Science Research Program through the National Research
Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2017R1D1A3B03031152).
Conflicts of Interest: The authors declare no conflicts of interest.
Sustainability 2019, 11, 3597 18 of 19

References
1. Roper, A.T.; Cunningham, S.W.; Porter, A.L.; Mason, T.W.; Rossini, F.A.; Banks, J. Forecasting and Management
of Technology, 2nd ed.; John Wiley & Sons: Hoboken, NJ, USA, 2011.
2. Jun, S. Bayesian Count Data Modeling for Finding Technological Sustainability. Sustainability 2018, 10, 3220.
[CrossRef]
3. Kim, J.; Jun, S. Graphical causal inference and copula regression model for Apple keywords by text mining.
Adv. Eng. Inf. 2015, 29, 918–929. [CrossRef]
4. Lee, J.; Kang, J.; Jun, S.; Lim, H.; Jang, D.; Park, S. Ensemble Modeling for Sustainable Technology Transfer.
Sustainability 2018, 10, 2278. [CrossRef]
5. Park, S.; Lee, S.; Jun, S. A network analysis model for selecting sustainable technology. Sustainability 2015, 7,
13126–13141. [CrossRef]
6. Hunt, D.; Nguyen, L.; Rodgers, M. Patent Searching Tools & Techniques; Wiley: Hoboken, NJ, USA, 2007.
7. Lee, B.; Won, D.; Park, J.; Kwon, L.; Moon, Y.; Kim, H. Patent-Enhancing Strategies by Industry in Korea
Using a Data Envelopment Analysis. Sustainability 2016, 8, 901. [CrossRef]
8. Chames, A.; Cooper, W.W.; Rhodes, E. Measuring the efficiency of decision making units. Eur. J. Oper. Res.
1978, 2, 429–444.
9. Forsund, F.R.; Sarafoglou, N. On the origins of data envelopment analysis. J. Product. Anal. 2002, 17, 23–40.
[CrossRef]
10. Murphy, K.P. Machine Learning: A Probabilistic Perspective; MIT Press: Cambridge, MA, USA, 2012.
11. International Patent Classification (IPC); World Intellectual Property Organization. Available online:
http://www.wipo.int/classifications/ipc/en (accessed on 30 October 2018).
12. Choi, J.; Jun, S.; Park, S. A Patent Analysis for Sustainable Technology Management. Sustainability 2016, 8,
688. [CrossRef]
13. Kim, S.; Jang, D.; Jun, S.; Park, S. A Novel Forecasting Methodology for Sustainable Management of Defense
Technology. Sustainability 2015, 7, 16720–16736. [CrossRef]
14. Kim, J.; Jun, S.; Jang, D.; Park, S. Sustainable Technology Analysis of Artificial Intelligence Using Bayesian
and Social Network Models. Sustainability 2018, 10, 115. [CrossRef]
15. Park, S.; Jun, S. Statistical Technology Analysis for Competitive Sustainability of Three Dimensional Printing.
Sustainability 2017, 9, 1142. [CrossRef]
16. Uhm, D.; Ryu, J.; Jun, S. An Interval Estimation Method of Patent Keyword Data for Sustainable Technology
Forecasting. Sustainability 2017, 9, 2025. [CrossRef]
17. Jun, S.; Park, S. Examining Technological Innovation of Apple Using Patent Analysis. Ind. Manag. Data Syst.
2013, 113, 890–907. [CrossRef]
18. Marinho, J.F.M.A.; Laurencel, L.C.; Amaral, M.R.S. Implementing DEA models in the R program. In
Proceedings of the 11th International Conference on Data Envelopment Analysis, Samsun, Turkey, 28–30
June 2013; pp. 1–6.
19. Banker, R.D.; Cooper, W.W.; Seiford, L.; Thrall, M.; Zhu, J. Returns to Scale in different DEA models. Eur. J.
Oper. Res. 2004, 154, 345–362. [CrossRef]
20. The United States Patent and Trademark Office (USPTO). Available online: http://www.uspto.gov (accessed
on 1 November 2018).
21. WIPSON. WIPS Corporation. Available online: http://www.wipson.com (accessed on 15 October 2018).
22. Feinerer, I.; Hornik, K. Package ‘tm’ Ver. 0.7-5, Text Mining Package, CRAN of R Project. Available online:
https://cran.r-project.org/web/packages/tm/tm.pdf (accessed on 1 August 2018).
23. R Development Core Team. R: A Language and Environment for Statistical Computing; R Foundation for
Statistical Computing: Vienna, Austria. Available online: http://www.R-project.org (accessed on 1 July 2018).
24. Chames, A.; Cooper, W.W.; Golany, B.; Stutz, J. Foundations of Data Envelopment Analysis for
Pareto-Koopmans efficient empirical production functions. J. Econom. 1985, 30, 91–107.
25. Chames, A.; Cooper, W.W.; Thrall, R.M. A structure for classifying and characterizing efficiency and
inefficiency in Data Envelopment Analysis. J. Prod. Anal. 1991, 2, 197–237.
26. Chames, A.; Haag, S.; Jaska, P.; Semple, J. Sensitivity of efficiency classifications in the additive model of data
envelopment analysis. Int. J. Syst. Sci. 1992, 23, 789–798.
Sustainability 2019, 11, 3597 19 of 19

27. Chames, A.; Rousseau, J.J.; Semple, J.H. Sensitivity and stability of efficiency classifications in data
envelopment analysis. J. Prod. Anal. 1996, 7, 5–18.
28. Fried, H.O.; Lovell, C.A.K.; Schmidt, S.S. The Measurement of Productive Efficiency and Productivity Growth;
Oxford University Press: New York, NY, USA, 2008.
29. Luenberger, D.G. Benefit Functions and Duality. J. Math. Econ. 1992, 21, 461–486. [CrossRef]
30. Luenberger, D.G. New Optimality Principles for Economic Efficiency and Equilibrium. J. Optim. Theory Appl.
1992, 75, 221–264. [CrossRef]
31. Luenberger, D.G. Dual Pareto Efficiency. J. Econ. Theory 1994, 62, 70–85. [CrossRef]
32. Cheng, G.; Zervopoulos, P. A Generalized Directional Distance Function in Data Envelopment Analysis and
Its Application to a Cross-Country Measurement of Health Efficiency. Available online: http://mpra.ub.uni-
muenchen.de/42068 (accessed on 1 July 2018).
33. Chung, Y.H.; Färe, R.; Grosskopf, S. Productivity and Undesirable Outputs: A Directional Distance Function
Approach. J. Environ. Manag. 1997, 51, 229–240. [CrossRef]
34. Oh, D.H.; Suh, D. Package ‘nonparaeff’ Ver. 0.5-8, Nonparametric Methods for Measuring Efficiency and
Productivity Package, CRAN of R Project. Available online: https://cran.r-project.org/web/packages/tm/tm.
pdf (accessed on 1 August 2018).
35. Korea Intellectual Property Strategy Agency (KISTA). Available online: http://www.kista.or.kr (accessed on
16 November 2018).
36. Commandeur, J.J.F.; Koopman, S.J.; Ooms, M. Statistical Software for State Space Methods. J. Stat. Softw.
2011, 41. [CrossRef]
37. Petris, G.; Petrone, S. State Space Models in R. J. Stat. Softw. 2011, 41. [CrossRef]
38. Hyndman, R.; Athanasopoulos, G.; Bergmeir, C.; Caceres, G.; Chhay, L.; O’Hara-Wild, M.; Petropoulos, F.;
Razbash, S. Package ‘forecast’ Ver. 8.7, Forecasting Functions for Time Series and Linear Models, CRAN of R
Project. 2019. Available online: https://cran.r-project.org/web/packages/forecast/forecast.pdf (accessed on
1 May 2019).
39. Petris, G.; Gilks, W. Package ‘dlm’ Ver. 1.1-5, Bayesian and Likelihood Analysis of Dynamic Linear Model,
CRAN of R Project. 2019. Available online: https://cran.r-project.org/web/packages/dlm/dlm.pdf (accessed
on 30 December 2018).

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).