Research Policy 38 (2009) 1534–1544

Contents lists available at ScienceDirect

Research Policy
journal homepage: www.elsevier.com/locate/respol

Impact analysis for national R&D funding in science and technology using
quantification method II
Jin Young Choi, Jong Ha Lee, So Young Sohn ∗
Department of Information & Industrial Engineering, Yonsei University, 134 Shinchon-dong, Seoul 120-749, Republic of Korea

a r t i c l e i n f o a b s t r a c t

Article history: As the influences of science and technology on its national economy have increased, each country has
Received 5 December 2008 devised various funding programs for research and development (R&D) projects. Numerous studies have
Received in revised form 27 May 2009 been conducted to evaluate the performance of such R&D projects. In most studies, the performances
Accepted 14 September 2009
were measured in terms of an ordinal Likert scale, but they were treated as continuous variables. Much
Available online 8 October 2009
important information can be buried when a categorical Likert scale is treated as continuous variable. In
this paper, we treat Likert scales as categorical and apply quantification method II to analyze the relation-
Keywords:
ship between short- to mid-term performance factors and long-term impact factor of R&D projects. We
R&D
Funding
apply the proposed approach to the survey data obtained from the Science and Technology Promotion
Performance Fund in Korea. The results of this paper are expected to contribute to a better understanding of the impact
Impact of R&D funding.
Quantification © 2009 Elsevier B.V. All rights reserved.

1. Introduction
Science and technology are the most important motivators for
a country to improve and strengthen its national economy and
global competitiveness. Many countries have made an effort to
increase their levels of science and technology with R&D projects.
The Korean government has been conducting diverse R&D funding
programs since the 1980s. The most representative R&D funds sup-
ported by the Korean government are the following: the Industrial
Technology Development Fund (ITDF), the Application Technology
Development Fund (ATDF), and the Science and Technology Pro-
motion Fund (STPF). The ITDF has been managed by the Ministry
of Commerce, Industry, and Energy (MOCIE) since 1986, while the
ATDF and STPF have been managed by the Ministry of Informa-
tion and Communication (MIC) and the Ministry of Science and
Technology (MOST), respectively, since 1993.
Many studies have been conducted to measure the per-
formances of these funding schemes. Most measured some
pre-determined performance indicators in terms of Likert scales.
For instance, Sohn et al. (2007) proposed a SEM (structural equa-
tion model) in order to measure the performances of STPF using
the survey data obtained from the firms which had received R&D
funding support from MOST during 2000–2003. Like other existing
studies regarding R&D performance, they considered the responses
measured in terms of a Likert scale as continuous variables. Accord-
∗ Corresponding author. Tel.: +82 2 2123 4014; fax: +82 2 364 7807.
E-mail address: sohns@yonsei.ac.kr (S.Y. Sohn).
ing to Dittrich et al. (2007), a common problem with all methods
utilizing Likert responses is that they treat Likert responses as abso-
lute measurements, but this can be rather dubious, especially when
dealing with subjective self-assessments. In addition, such treat-
ment can overlook the possibility that respondents giving the same
answer to a Likert item neither share the same response value nor
are equivalent with respect to the attitudes or values to be mea-
sured. These problems may arise when the Likert responses are
treated as continuous rather than categorical. Furthermore, much
important information can be buried when a categorical Likert scale
is treated as a continuous variable.
In this study, we treat Likert scale responses typically obtained
for performance analyses for national R&D funding schemes as ordi-
nal categorical variables. According to Tanaka (1979) and Hayashi’s
(1954) quantification method II is suitable when predicting the
qualitative external criterion (response variables) on the basis of
the information concerning the qualitative attributes of each sub-
ject (predictor variables). In this study, all measurement variables
of each performance factor are qualitative observed in an ordinal
Likert scale. The performance factors of this study consist of out-
put, outcome, and impact. Output and outcome factors represent
the performances of the individual company level and they are
interpreted as short- to medium-term performances, while impact
factor represents the performance of the industrial and national
level and it is interpreted as long-term performance. Thus, we use
quantification method II to analyze the relationship between long-
term impact and short- to medium-term performances obtained
from companies which had national R&D funding. Furthermore,
we conduct cluster analysis contingency analysis to provide these

0048-7333/$ – see front matter © 2009 Elsevier B.V. All rights reserved.
doi:10.1016/j.respol.2009.09.005
 J.Y. Choi et al. / Research Policy 38 (2009) 1534–1544
companies with crucial feedback information for the innovative
performances of R&D funding.
This paper is organized in the following manner. Section 2
includes a literature review and relevant theories related to per-
formance analysis. In Section 3, the proposed analysis procedure is
introduced, along with the case that motivated this study. In Section
4, the proposed quantification method II is applied to a real case. In
addition, cluster analysis and contingency analysis are applied to
the result of quantification analysis in order to explore the features
of grouped firms which have similar characteristics. In the last sec-
tion, our study results are discussed and concluding remarks are
offered.
2. Literature review
2.1. R&D performance measures
The measures of R&D performance are classified as quantita-
tive, semi-quantitative, and qualitative (Schumann et al., 1995;
Werner and Souder, 1997; Kerssens-van Drongelen et al., 2000).
Quantitative measures generate numbers that closely represent
specific areas of R&D activity; examples include the percentage of
proposals won, the number of tests performed per week, and the
return on investment (ROI) (Brown and Gobeli, 1992; Kerssens-van
Drongelen et al., 2000). Qualitative measures are essentially those
based on personal or subjective judgments; they often encom-
pass larger and relatively important dimensions, like the originality
of an idea, and marketplace potential (Brown and Gobeli, 1992).
Sometimes, qualitative values are converted into a numerical value
by anchoring the qualitative indicators to definitions or behav-
iors that can be expressed as numerical equivalents, such as a
five-point scale. Measurement methods that use such conversion
of judgments into numerical values are often referred to by the
term semi-quantitative (Brown and Gobeli, 1992; Kerssens-van
Drongelen et al., 2000; Pappas and Remer, 1985).
In general, a R&D performance analysis model uses at least one
of these three types of measures, or uses a combination of them.
Small and Swann (1993) used quantitative measures, such as R&D
expenditure, R&D/sales ratios, and R&D per employee, in order to
measure and compare the R&D performance of British companies
with those of companies in the USA, Germany, and Japan. Coccia
(2001) developed a mathematical model measuring the perfor-
mance of R&D organizations in Italy based on an analysis of R&D
activities with quantitative measures, such as the amounts of funds,
the number of personnel in training and courses, international pub-
lications, and patents. Revilla et al. (2003) provided an overview
of how a data envelopment analysis (DEA) approach was used to
investigate some characteristics of performance for joint R&D col-
laboration projects. In this study, the inputs were total revenue,
number of employees in the company, and total R&D budget; the
outputs were total number of patents, total employment, and total
income generated by the concerted project. Both the inputs and
outputs were used as quantitative measures in their study.
Among the studies investigated during our literature review,
there is no case using only qualitative measures for R&D perfor-
mance analysis. Not surprisingly, most R&D performance analyses
used mixed measures, which are combinations of quantitative,
qualitative, and/or semi-quantitative measures. The integrated
metrics that combine several types of quantitative and qualita-
tive measures can best demonstrate the perceived cost/benefit of
each alternative (Werner and Souder, 1997). Bremser and Barsky
(2004) investigated how the BSC (Balanced Scorecard) is use-
ful for seven performance measurement functions as they relate
to R&D. In their study, the authors presented both quantitative
and qualitative measures, such as number of patents awarded
1535
(quantitative) and customer satisfaction with new products (qual-
itative).
Meanwhile, there has been a recent increase in the number
of studies using semi-quantitative measures of R&D performance,
especially using Likert scales. Belderbos et al. (2004) analyzed
the impact of R&D cooperation on firm performance, differen-
tiating among four types of R&D partners and considering two
performance measures: labour productivity and productivity in
innovative sales. For this, they used the CIS (Community Innovation
Surveys), which asks each firm to rate the importance of various
external sources of information on a Likert scale (1–5).
According to Pappas and Remer (1985), semi-quantitative mea-
sures are the best because they deal with R&D outputs more flexibly
than either quantitative or purely qualitative methods. However,
Pappas and Remer (1985) also indicated that using numbered scales
can always be misleading since some people believe there can never
be a perfect ‘10’ while others think everything is perfect. Thus, rela-
tive differences in qualitative scaling can distort the output if these
measures are incorrectly combined. In the current study, we use
quantification method II to avoid such problems when interpreting
Likert scale data on R&D performance analysis.
2.2. Quantification method II
In some experimental and observational studies, the responses
and attributes of subjects are measured only by qualitative
categories. In order to analyze such observations, methods of quan-
tification or optimal scaling have been proposed by Fisher (1946),
Guttman (1941), and Hayashi (1950, 1952, 1954), among others.
According to these methods, scores are assigned optimally to these
qualitative categories in some objective and operational sense
(Tanaka, 1979). Among those theories for quantification or opti-
mal scaling, we focused on Hayashi’s quantification method II in
the present paper.
Hayashi (1950, 1952) treated some methods of quantification
of qualitative data in multidimensional analysis, especially the use
of quantification of qualitative patterns to secure the maximum
success rate of the prediction of phenomena from the statistical
point of view. According to Hayashi (1954), quantification means
that the patterns are categorized and given numerical values in
order that they may be treated as several indices, while classifica-
tion means a prediction of phenomena. In addition, quantification
is not used for finding numerical values, but for assigning them to
patterns from the operational point of view. According to Tanaka’s
(1978, 1979) definition, quantification method II is a way to pre-
dict the qualitative external criterion on the basis of information
concerning the qualitative attributes of each subject, and to ana-
lyze the influence of each attribute to the discrimination of the
external criterion. In the review paper of the quantification meth-
ods, Tanaka (1979) additionally defined ‘external criterion’, which
is also called ‘outside variable’, to mean ‘something to be predicted
or explained’. The author also explained that contribution of each
attribute to the external criterion is measured by the partial cor-
relation coefficient or approximately by the range of the assigned
scores; the range is calculated by subjecting minimum of quantifi-
cation score from maximum of quantification score. In the same
paper, the author pointed out that Hayashi’s second method may
be applied in the following two different manners. One is the case
where a response item is chosen as the external criterion and the
problem is to predict the response from the qualitative factors by a
similar way as the regression analysis. The other is the case where a
factor is chosen as the external criterion and the problem is to dis-
criminate between groups corresponding to the categories of the
chosen factor by a similar way as the canonical correlation analy-
sis. The method used in the current study is the latter case which
is much closer to canonical correlation analysis.
1536 J.Y. Choi et al. / Research Policy 38 (2009) 1534–1544

Table 1
Factors and measurement variables.

Factors Measurement variables Variable names Frequency of measurement variable value

1 2 3 4 5

Technology performance Improvement of a technological ability X1 1 0 24 123 25

Technological progress X2 0 6 31 64 72
Conquest of a technological gap X3 0 1 22 108 41
Localization of the technology X4 0 0 28 108 42

Business performance Improvement of business ability X5 1 2 52 102 16

Improvement of a company’s popularity X6 0 1 34 109 29
Improvement of market share X7 0 3 48 109 13
Sales and export increase X8 1 1 47 106 18

Management performance Improvement of employment X9 1 4 47 115 6

Improvement of R&D environment X10 1 1 26 130 15
Improvement of manufacturing environment X11 1 1 57 104 10
Improvement of management information systems X12 1 1 50 116 5

Manufacturing performance Reduction of product defect X13 0 1 31 131 10

Improvement of productivity X14 0 2 39 120 12
Reduction of manufacturing period X15 0 2 54 109 8
Reduction of manufacturing cost X16 0 4 119 44 6

Impact Improvement of science & technology Y1 1 0 40 118 14

Industrial development Y2 1 0 30 120 22
Increase of national demand Y3 1 1 37 115 19
Improvement of industrial structure Y4 0 2 53 109 9
Improvement of national economy Y5 0 2 45 115 11
Improvement of global competitiveness Y6 0 2 37 118 16

According to Suzuki and Kudof (1979), Hayashi’s quantification
method II is a method of multivariate discrimination analysis to
manipulate attribute data as predictor variables, and it is very useful
in the medical research field for the estimation, diagnosis, progno-
sis, and evaluation of epidemiological factors, and other problems
based on the multiplicity of attribute data.
3. Research data and procedure
3.1. Motivating case and data
This study used the survey data collected in a study by Sohn et al.
(2007). In the previous study, Sohn et al. (2007) performed a survey
of the companies supported by Science and Technology Promotion
Fund during 2000–2003 in order to evaluate the effects of national
funding on R&D projects. The contents of the survey comprised
not only the characteristics of a R&D project and company, but also
measurement variables for performance factors. All of the variables
used in the survey were measured using a five-point Likert scale,
with a higher number representing better performance. Table 1
shows the short- to mid-term performance factors and the impact
factor, along with the measurement variables of each factor almost
used in the current study. In the previous study, they were obtained
based on 173 surveys returned. From these results of the survey,
the factors associated with R&D performance, such as technology,
business, management, and manufacturing performances, as well
as the impact factor, were analyzed in the previous study.
We have adopted such five factors and most measurement vari-
ables of previous study. However, some measurement variables
in business, manufacturing performance and impact factor were
newly included in the current study. Specifically, ‘Improvement
of market share’ was included in business performance because
market-share position widely believed to be a determinant of
business performance (Venkatraman, 1986). ‘Reduction of prod-
uct defect’, ‘Reduction of manufacturing period’, and ‘Reduction of
manufacturing cost’ were included in manufacturing performance
because those variables have been considered as crucial measures
in many researches on measuring manufacturing performance
(Kaplan, 1983; Jonsson and Lesshammar, 1999). ‘Improvement of
global competitiveness’ was included in impact factor because
‘global competitiveness’ can be a key indicator to measure long-
term industrial and national growth.
In addition to changes of measurement variables, our analysis is
apparently different from the previous study in terms of the follow-
ing two aspects. First, we regrouped output and outcome into a set
of short- to mid-term performance factors and tried to see its rela-
tionship with the impact factor. In the previous study, the authors
classified R&D fund recipient’s performance into three aspects:
output, outcome and impact factor. The authors also categorized
technological performance into output, business, management and
manufacturing performances into outcome, and long-term impact
factor into impact. However, since the output and outcome factors
indicate the direct achievements yield from R&D project, whereas
impact factor presents indirect achievements and its effects takes
a long time to be appeared, we combine both output and outcome
into a set of short- to mid-term performance. Moreover, in both
present and current study, output and outcome represent the per-
formances of the individual company level, and impact represents
the performance of the industrial and national level. In these senses,
we analyze the relationship between a set of short- to mid-term
performance factors and the impact factor.
The second aspect is that we use the quantification method to
identify non-linear patternship between the two groups of perfor-
mance factors, whereas Sohn et al. used the structural equation
model to observe linear patternship between performance factors.
3.2. Procedure
The procedure of this study consists of two steps: in step 1,
quantification analysis is performed to investigate the relation-
ship between a set of short- to mid-term performance factors and
the impact factor. The relationships between the two sets of vari-
ables, more precisely, the measurement variables pertaining to a
set of short- to mid-term performance factors and the impact fac-
tor, are also investigated in this step. In step 2, cluster analysis was
performed using the results from step 1, in order to present the
features of grouped firms which have similar characteristics. The
entire framework for the current study is displayed in Fig. 1.
 J.Y. Choi et al. / Research Policy 38 (2009) 1534–1544
Table 2
The results of canonical correlation analysis.
Variate number Canonical correlation R2 Eigenvalue
1 0.7548 0.5697 1.3237
2 0.7466 0.5574 1.2594
3 0.6233 0.3885 0.6354
4. Statistical analysis
4.1. Quantification analysis
To analyze the performance of the Science and Technology Pro-
motion Fund, we assumed that there would be some relationships
between specific performances, such as technology, business, man-
agement, and manufacturing performance indices. Furthermore,
we expected that there would be some relationships between
long-term impact factor of R&D projects and short- to mid-term
performance factors. Thus, the external criterion in this study is
long-term impact factor that represents the performance of the
industrial and national level generally caused by short- to mid-term
performance factors (Sohn et al., 2007). Quantification method II is
performed in this section to verify the relationships among their
performance factors.
While numerous existing studies using Likert responses treat
survey items (measurement variables) as continuous, the present
study treats them as categorical. For this purpose, all measurement
variables were converted into dummy variables. The data used in
this study is rated on a five-point Likert scale, so each measurement
variable was divided and transformed into five kinds of dummy
variables (see Table 1). For each survey item, a value of ‘1’ means
that the change after conducting a R&D project is ‘very deteriora-
tive’, ‘2’ means ‘deteriorative’, ‘3’ means ‘not changed’, ‘4’ means
‘ameliorative’, and ‘5’ means ‘very ameliorative’.
However, as shown in Table 1, the frequency of survey data cate-
gorized as ‘very deteriorative’ and ‘deteriorative’ was not sufficient
for analysis, and this made it difficult to apply the quantification
method. Moreover, when we performed the canonical analysis with
five kinds of dummy variables, some estimates of raw canoni-
cal coefficient were ‘0’ because some dummy variables, such as
‘technological progress’ and ‘conquest of a technological gap’, were
not observed. To overcome this problem, the five-level measure-
ment scale was redesigned and regrouped into three levels. In
other words, ‘very deteriorative’ and ‘deteriorative’, lacking enough
frequency, were bound together into one level. ‘Not changed’
also joined that level. Because the purpose of the survey was
to investigate how ameliorative the R&D performance was after
the R&D project occurred, those three categories, ‘very deteriora-
tive’, ‘deteriorative’ and ‘not changed’, could be grouped into the
category ‘not changed or deteriorative’. Other dummy variables,
‘ameliorative’ and ‘very ameliorative’, were retained for the same
reason. Hence, the changed levels of each measurement variable
are:
Fig. 1. Framework for current study.
1537
Cumulative Wilks’ lambda Approx F Degree of freedom P-value
0.2435 0.0171 1.69 384 <0.0001
0.4752 0.0397 1.48 341 <0.0001
0.5921 0.0896 1.23 300 0.0107
level 1 = ‘not changed or deteriorative’;
level 2 = ‘ameliorative’;
level 3 = ‘very ameliorative’.
For example, dummy variables of ‘Improvement of a techno-
logical ability (X1)’ were designated as X1 1, X1 2 and X1 3. X1 1
indicated that R&D project performance with respect to ‘Improve-
ment of a technological ability’ was ‘not changed or deteriorative’,
X1 2 indicated ‘ameliorative’, and X1 3 indicated ‘very ameliora-
tive’. The quantification analysis was performed with those three
kinds of dummy variables for each measurement variable.
Firstly, the canonical correlation analysis (CCA) was conducted
with those dummy variables; dummy variables of measurement
variables pertaining to technology, business, management, and
manufacturing performance factors were used as one set of
variables (X), while dummy variables of measurement variables
pertaining to the impact factor were used as another set of vari-
ables (Y). Table 2 shows the result of canonical correlation analysis
with dummy variables.
Throughout canonical correlation analysis, a total of 18 pairs of
canonical variates were yielded. Of these 18, only three pairs of
canonical variates were statistically significant (P-value < 0.05, see
Table 2). The first, second, and third pairs of canonical variates con-
sisted of three kinds of axes, and those three axes explain 59% of
the total survey data (first axis: 24.35%, second axis: 23.17%, third
axis: 11.69%). In the following step, quantification analysis was con-
ducted underlying those three axes. Table 3 shows the results of
quantification for a set of short- to mid-term performance factors
at the first axis. The results of quantification for the impact factor
at the first axis are given in Table 4.
To verify which performance factor most significantly affects
impact, the sum of the ranges for measurement variables pertain-
ing to each factor was computed. For instance, the magnitude of
the influence of technology performance toward impact was cal-
culated as the total of the ranges for X1, X2, X3 and X4. In this
manner, the sum of the ranges for all factors was yielded in the
following order: manufacturing performance (4.66), management
performance (3.99), business performance (2.45) and technology
performance (1.79). From this, it can be verified that manufactur-
ing performance most significantly affects the impact. In the case
of the second axis, although the first and second axes were theo-
retically orthogonal and independent, manufacturing performance
was again the most influential factor on impact, as it was in the
case of the first axis: manufacturing performance (5.17), business
performance (3.07), management performance (2.90) and tech-
1538 J.Y. Choi et al. / Research Policy 38 (2009) 1534–1544

Table 3
The results of quantification for a set of short- to mid-term performance factors at the first axis.

Factors Measurement variables Dummy variables Raw coefficient Scores Range Patterns

Technology performance Improvement of a technological X1 1 0.00 0.25 0.34

ability (X1) X1 2 −0.34 −0.09 Negative and positive
X1 3 −0.07 0.18
Technological progress (X2) X2 1 0.00 −0.25 0.48
X2 2 0.48 0.23 Positive and negative
X2 3 0.18 −0.07
Conquest of a technological gap X3 1 0.00 0.01 0.47
(X3) X3 2 0.12 0.13 Positive and negative
X3 3 −0.35 −0.34
Localization of the technology (X4) X4 1 0.00 −0.30 0.50
X4 2 0.28 −0.02 Positive
X4 3 0.50 0.20

Business performance Improvement of business ability X5 1 0.00 0.18 0.71

(X5) X5 2 −0.19 −0.01 Negative
X5 3 −0.71 −0.54
Improvement of a company’s X6 1 0.00 −0.50 1.02
popularity (X6) X6 2 0.52 0.02 Positive
X6 3 1.02 0.52
Improvement of market share (X7) X7 1 0.00 0.05 0.17
X7 2 −0.06 −0.01 Negative
X7 3 −0.17 −0.12
Sales and export increase (X8) X8 1 0.00 −0.07 0.55
X8 2 0.17 0.11 Positive and negative
X8 3 −0.38 −0.44

Management performance Improvement of employment (X9) X9 1 0.00 0.01 0.81

X9 2 0.02 0.04 Positive and negative
X9 3 −0.79 −0.78
Improvement of R&D environment X10 1 0.00 −0.52 2.31
(X10) X10 2 0.43 −0.09 Positive
X10 3 2.31 1.79
Improvement of manufacturing X11 1 0.00 −0.02 0.11
environment (X11) X11 2 0.03 0.00 Positive
X11 3 0.11 0.09
Improvement of MIS (X12) X12 1 0.00 −0.35 0.75
X12 2 0.54 0.18 Positive and negative
X12 3 −0.22 −0.57

Manufacturing performance Reduction of product defect (X13) X13 1 0.00 0.42 0.85
X13 2 −0.58 −0.16 Negative and positive
X13 3 0.28 0.70
Improvement of productivity (X14) X14 1 0.00 −0.20 0.65
X14 2 0.22 0.02 Positive
X14 3 0.65 0.45
Reduction of manufacturing period X15 1 0.00 −0.10 2.20
(X15) X15 2 0.30 0.20 Positive and negative
X15 3 −1.90 −2.00
Reduction of manufacturing cost X16 1 0.00 −0.14 0.96
(X16) X16 2 0.43 0.29 Positive
X16 3 0.96 0.81

Table 4
The results of quantification for the impact factor at the first axis.

Factors Measurement variables Dummy variables Raw coefficient Scores Range Patterns

Impact Improvement of science and Y1 1 0.00 0.25 0.61

technology (Y1) Y1 2 −0.08 0.17 Negative and positive
Y1 3 0.53 0.78
Industrial development (Y2) Y2 1 0.00 −0.25 1.98
Y2 2 0.25 −0.01 Positive
Y2 3 1.98 1.73
Increase of national demand (Y3) Y3 1 0.00 0.01 0.85
Y3 2 0.54 0.55 Positive and negative
Y3 3 −0.31 −0.30
Improvement of industrial Y4 1 0.00 −0.30 1.48
structure (Y4) Y4 2 −0.14 −0.44 Negative
Y4 3 −1.48 −1.78
Improvement of national economy Y5 1 0.00 0.18 1.40
(Y5) Y5 2 0.52 0.70 Positive and negative
Y5 3 −0.88 −0.70
Improvement of global Y6 1 0.00 −0.50 1.84
competitiveness (Y6) Y6 2 0.75 0.25 Positive
Y6 3 1.84 1.34
 J.Y. Choi et al. / Research Policy 38 (2009) 1534–1544
nology performance (1.76). In the case of the third axis, business
performance was the most influential factor on impact: business
performance (7.07), management performance (4.06), manufac-
turing performance (3.67) and technology performance (2.00). The
results of quantification for the second and third axes are included
in the Appendix.
In quantification analysis, the range for each short- to mid-term
performance factor represents the magnitude of its influence on the
impact factor, while the sign of the quantification score indicates
a direction, moving from lower level to higher level. If the sign of
the quantification score for a particular measurement variable has
a positive direction (moving from (−) to (+) as the level of the vari-
able increases), for example from level 1 to level 3, while another
variable has the same moving direction pattern, the two variables
are positively correlated. In particular, when the range size of a
measurement variable is large, there would be a stronger relation-
ship. Hence, among measurement variables having a wide range
of values, those which have a consistent direction show a highly
influential relationship on each other. Usually, the quantification
scores show a linear pattern in the same measurement variables.
However, a non-linear pattern between quantification scores in
the same measurement variable was detected in this study. Those
patterns were very unusual, but it was found that they have similar
patterns, such as ‘positive and negative’ or ‘negative and positive’.
The ‘positive’ or ‘negative’ patterns showed a linear pattern where
the quantification scores increased in a direction that is consis-
tent with the change in the level of the dummy variable. On the
other hand, ‘positive and negative’ or ‘negative and positive’ pat-
terns showed non-linear patterns in which the direction of the sign
changed as the level of the dummy variable increased.
For example, the quantification scores of ‘localization of the
technology (X4)’ variables increase consistently as the level of
dummy variables increases. However, in the case of ‘Improvement
of a technological ability (X1)’, in the first interval, X1 1 to X1 2,
the quantification score decreases, but in the second interval, X1 2
to X1 3, the quantification score increases. The variables with such
patterns were designated as having a ‘negative and positive’ pat-
tern. In the same manner, quantification scores for each short- to
mid-term performance factor are expressed as a specific pattern
of the four patterns mentioned above. The patterns for each short-
to mid-term performance factor and impact factor are described in
Tables 3 and 4, respectively.
Based on the patterns of direction for quantification scores,
the relationships between variables having similar patterns were
investigated. In the first axis, ‘Improvement of R&D environ-
ment (2.31)’ and ‘Reduction of manufacturing period (2.20)’ have
larger value ranges than other short- to mid-term performance
factor variables. For the impact factor, the variables ‘Industrial
development (1.98)’ and ‘Improvement of global competitiveness
(1.84)’ have larger value ranges than other variables. Thus, these
four variables affect each other most significantly. The pattern of
‘Improvement of R&D environment (X10)’ shows a phase increas-
ing incrementally in the positive direction as the level of the dummy
variables increase; this is very similar to the patterns of ‘Industrial
development (Y2)’ and ‘Improvement of global competitiveness
(Y6)’. From this, it is possible to interpret that the more ameliora-
tive a R&D environment is, the greater the Industrial development
and global competitiveness. The inverse interpretation can also
be possible because those variables have the same direction for
quantification scores; for example, the more deteriorative a R&D
environment is, the lower the Industrial development and global
competitiveness.
In the case of the pattern of ‘Reduction of manufacturing period
(X15)’, the quantification score increases in the positive direction
until reaching level 2, but from level 2 to level 3, the score decreases
in the negative direction. It is an exclusively different pattern com-
1539
pared with the three variables mentioned above. Such a result
can be explained as follows: in the first interval, level 1 to level
2, the quantification scores of ‘Reduction of manufacturing period
(X15)’ increase in the same direction as those of ‘Industrial devel-
opment (Y2)’ and ‘Improvement of global competitiveness (Y6)’;
thus, the larger the reduction of manufacturing period, the higher
the Industrial development and global competitiveness. However,
in the second interval, level 2 to level 3, it was found that indus-
trial development and global competitiveness still increase, even
though the reduction of manufacturing period decreases in the
negative direction. This can be interpreted as follows: if industrial
development and global competitiveness reaches at a stable point,
the reduction of manufacturing period can no longer have a positive
impact on them because the manufacturing period has a marginal
point to be reduced. Thus, we conclude that the reduction of man-
ufacturing period time is clearly essential factor for increasing both
industrial development and global competitiveness, however, after
certain period of time, the industrial development and global com-
petitiveness last its continuity to be increased regardless of the
manufacturing period reduction.
The interpretation of the quantification result for the second
axis was conducted in the same manner of analysis as for the
first axis. The independent variable having the largest range was
‘Reduction of manufacturing period (2.66)’. The impact factor hav-
ing the largest range was ‘Increase of national demand (2.95)’. Both
of these two variables, ‘Reduction of manufacturing period (X15)’
and ‘Increase of national demand (Y3)’, have the same ‘positive’
pattern. Thus, it can be interpreted that the higher the reduction of
manufacturing period, the higher the national demand. The inverse
interpretation is also possible; the lower the reduction of manufac-
turing period, the lower the national demand.
In the quantification result for the third axis, a set of short-
to mid-term performance factors having the largest range were
‘Improvement of business ability (2.06)’ and ‘Reduction of manufac-
turing period (2.51)’. For the impact factor, ‘Improvement of science
and technology (2.25)’ and ‘Improvement of industrial structure
(2.05)’ were the variables having the largest ranges. In terms of pat-
terns for those variables, ‘Improvement of business ability (X5)’ and
‘Improvement of industrial structure (Y4)’ both have a ‘negative and
positive’ pattern. ‘Reduction of manufacturing period (X15)’ and
‘Improvement of science and technology (Y1)’ both have a ‘positive’
pattern. A possible interpretation is that the greater the business
ability of firms, the better their industrial structure. The inverse
is also possible. Similarly, it is possible to interpret that the higher
the reduction of the manufacturing period, the higher the improve-
ment of science and technology. The inverse interpretation can also
be possible because the directions of quantification scores for those
two variables are the same.
4.2. Clustering analysis
In this section, the cluster analysis was conducted on the basis
of the scores of each performance factor calculated with the ranges
derived from quantification analysis. This work is meaningful
because it shows us how we utilize the results of quantification
analysis. By grouping the companies with their performance scores
in view of four performance factors, such as, technology, busi-
ness, management and manufacturing performances, we can detect
which performances of specific group are weak or strong. Since the
ranges represent a relative importance toward long-term impact,
the results will show what the specific group should do in order
to improve overall long-term impact. That is an important issue
for the R&D policy planners who are intended to manage and
maintain the level of long-term impact. In the cluster analysis, we
focus on four performance factors of predictor variables, not impact
factor, because the impact factor is the result caused by four perfor-
1540 J.Y. Choi et al. / Research Policy 38 (2009) 1534–1544

Based on the results of cluster analysis, the characteristics of

each cluster were found as follows:

Cluster 1: this group shows poor performances with respect to all

Fig. 2. R2 (%) and pseudo-F by the number of cluster.
mance factors, however, we additionally provide the score of the
impact factor for each cluster to show the characteristics of each
cluster.
To calculate the scores of each performance factor, we use
the ranges derived from quantification analysis as weights for
each measurement variables (see Table 3). The value of each per-
formance factor, such as technology, business, management, and
manufacturing factor (i ) is a weighted sum of measurement vari-
ables (Xij ) with the weight (Wij ):

n
i = Wij Xij
j=1
where i represents the factor (1, 2, . . ., 4) and j represents the mea-
surement variable for each factor i.
Using the value of each performance factor, the performance
score for a specific company (Si ) was calculated in the following
manner:
S[i ] − Min[i ]
Si = × 100
Max[i ] − Min[i ]
where S[i ] is the value of each performance factor for a specific
company S, Max[i ] is the maximum value for performance factor i,
and Min[i ] is the minimum value for the performance factor i over
all companies.
We use K-Means algorithm for the cluster analysis; the K-Means
Method is an algorithm to classify the entire data set into relatively
similar K clusters by allocating each item in the nearest cluster
based on the center of chosen K-clusters. The number of clusters
was determined by considering R2 and pseudo-F values based on
the variation of cluster numbers. Generally, the number of clusters
is determined by considering the change of R2 (the proportion of
variance accounted for by the clusters) or a relatively large pseudo-
F value which indicates a stopping point (Eder et al., 1994; Lattin
et al., 2002). Therefore, we selected four clusters as the number of
clusters on the basis of a strong increment in R2 and a strong decre-
ment in pseudo-F value. The graph of the R2 and pseudo-F values
is also given in Fig. 2. Based on the results in Fig. 2, we select four
cluster numbers.
Table 5 is a result of cluster analysis and shows the average per-
formance scores of each cluster as well as the overall average of all
clusters.
four factors. All performance scores were much lower than overall
average of all clusters. Thus the company which belongs to this
group should make an effort to lift up all aspects of short- and
mid-term performances to increase impact factor.
Cluster 2: this group has distinguished performance in technol-
ogy, management, and manufacturing. In particular, this group has
outstanding performance in technology. Even though the business
performance is not much higher than overall average of all clus-
ters, it is still higher than the average. Thus, this group is the best
performing group among all four groups.
Cluster 3: all average performance scores of this group are higher
than overall average of all clusters, except for technology per-
formance. Thus, all companies in this group should pay more
attention to improving technology performance.
Cluster 4: this group has distinguished performance in technol-
ogy and business. This group performs on average in terms of
management and manufacturing.
In summary, three clusters except for cluster 1 show well perfor-
mance in terms of four performance factors. The cluster 1 is the only
cluster which has very poor performance than others. The average
(1) impact factor score of each cluster is also given in Table 5 to show
how significant the relationships between four factor scores of each
cluster and their impact factor. This results support our expectation
that the companies in the cluster having higher score in four per-
formance factor may have higher score in impact score. As seen in
Table 5, the score of impact factor for cluster 2, 3, and 4 is much
higher than cluster 1.
Additionally, the contingency analysis was employed to grasp
(2) the additional characteristics of each cluster. Table 6 shows the
variables used for contingency analysis and the statistic results.
According to contingency analysis, three explanatory variables,
‘The location of headquarters’, ‘Presence or absence of research
department’, and ‘Life cycle of product’, are significantly associ-
ated with clusters at the 5% confidence level. The results of the
contingency analysis are displayed in Tables 7.1 and 7.2.
Additionally, cluster 2 has relatively good performance,
especially in technology performance, and its ratio of research
department-holding (58%) is higher than other clusters. This result
shows that the companies in possession of a research department
perform better than those without. Cluster 1 is the only case that
has a decline in life cycle; this may possibly contribute to its lower
performance. In comparison to cluster 1, clusters 2, 3, and 4 have
a larger portion of non-capital located companies and yield good
performance; the regional industrial development policy of the
Korean government is thought to be responsible for this finding (Oh,
2002).
Based on clustering and contingency analysis, we could see the
characteristics of both companies that perform well and poorly in
terms of four performance factors. However, unless both companies
that perform well and those that perform poorly are investigated
further, reasons for differences in performance will remain unclear.

Table 5
The results of cluster analysis.

Cluster Frequency Technology performance Business performance Management performance Manufacturing performance Impact

1 40 31.88 15.04 15.40 11.11 24.92

2 45 80.19 46.72 54.92 51.69 48.65
3 61 44.54 45.34 47.61 36.77 45.24
4 27 78.54 74.30 48.17 33.27 52.06
Average 56.18 43.22 42.15 34.17 42.49
 J.Y. Choi et al. / Research Policy 38 (2009) 1534–1544 1541

Table 6
Variables for contingency analysis and P-value.

Category Variable Type P-value

The location of headquarters Binary (capital, non-capital) 0.03*

The type of business Nominal (IT, BT, NT, ST, ET, CT) 0.26
Life cycle of product Nominal (introduction, growth, maturity, decline) 0.02*
Explanatory variables
The type of loan warrant Binary (general warrant or technology warrant) 0.64
Presence or absence of research department Binary 0.02*
Collaboration with foreign research institute or not Binary 0.55
Dependent variable Four clusters Nominal
*
Significant at 5%.

Table 7.1
The result of contingency analysis.

Presence or absence of research department The location of headquarters

0 (Absence) 1 (Presence) 0 (Non-capital) 1 (Capital)

Cluster 1 27(68%) 13(32%) 40(26%) Cluster 1 8(20%) 32(80%) 40(24%)

Cluster 2 19(42%) 26(58%) 45(29%) Cluster 2 20(44%) 25(55%) 45(27%)
Cluster 3 28(70%) 12(30%) 40(26%) Cluster 3 23(43%) 31(57%) 54(33%)
Cluster 4 19(70%) 8(30%) 27(17%) Cluster 4 14(52%) 13(48%) 27(16%)
93(61%) 59(39%) 152(100%) 65(39%) 101(61%) 166(100%)

Table 7.2 2. From the quantification analysis, it was found that ‘Reduction
The result of contingency analysis.
of manufacturing period’ was the most influential variable on
Life cycle of product the impact factor in terms of ‘Industrial development’, ‘Improve-
ment of global competitiveness’ and ‘Increase of national
Introduction Growth Maturity Decline
demand’. In addition, it was also found that ‘Improvement of R&D
Cluster 1 12(30%) 16(41%) 8(20%) 3(8%) 39(23%)
environment’ affects ‘Industrial development’ and ‘Improve-
Cluster 2 20(44%) 19(42%) 6(13%) 0 45(27%)
Cluster 3 20(35%) 32(56%) 5(9%) 0 57(34%) ment of global competitiveness’, while ‘Improvement of business
Cluster 4 10(37%) 8(31%) 8(31%) 0 26(16%) ability’ affects ‘Improvement of industrial structure’. These find-
61(37%) 75(45%) 27(16%) 3(2%) 167(100%) ings imply the following: when the government or the firm
conducts a R&D project, it should focus on both manufactur-
ing and management factors to increase the impact factor. In
Nonetheless, our analysis indicates that approximately 76.88% of detail, the R&D policy planner should concentrate on ‘Reduction
the companies, comprising those which belong to clusters 2, 3, and of manufacturing period’, ‘Improvement of R&D environment’
4 supported by national R&D funding have improved performance. and ‘Improvement of business ability’ areas.
3. By grouping the companies with their performance scores in the
view of four performance factors, such as, technology, business,
5. Conclusions management and manufacturing performances, we detected
which performances of specific group are weak or strong. Among
As greater attention has been paid to government R&D fund- four clusters, three clusters except for cluster 1 show good per-
ing programs, a variety of attempts have been made for effective formance in terms of four performance factors. Throughout the
performance assessment. In this sense, the performance of govern- results of clustering analysis, we suggest that the companies
ment R&D funding programs was analyzed in the current study which belong to cluster 1 showing poor performances in over-
using statistical methods, such as quantification method II and all four performance factors should make an effort to lift up all
cluster analysis. Quantification method II was used to analyze the aspects of short- mid- term performances to increase impact
relationship between a set of short- to mid-term performance factor. Moreover, R&D planners and practitioners should con-
factors and long-term impact factor of R&D projects and cluster sider which cluster their companies are positioned so that they
analysis was used to explore the features of firms which have sim- can provide proper advice or monitor the performance of the
ilar characteristics. companies.
Several findings were derived from the present study, as follows: 4. According to contingency analysis, three explanatory variables,
‘The location of headquarters’, ‘Presence or absence of research
1. Unlike existing studies, the current study treated Likert department’, and ‘Life cycle of product’, are significantly associ-
responses as categorical, using quantification analysis. A remark- ated with clusters at the 5% confidence level. This result shows
able aspect of the analysis process used in the current study is that the companies in possession of a research department per-
that the patterns of direction for quantification scores were used form better than those without. By the results of cluster analysis,
in the quantification analysis. In particular, non-linear pattern, we found that cluster 1 is the only case that has a decline in life
for example, negative direction from level 1 to 2 and positive cycle and we inferred that this may possibly contribute to its
direction from level 2 to 3, such as ‘Reduction of manufacturing lower performance.
period’, was detected, which was impossible to be observed in 5. Based on the results of both clustering and contingency analy-
the SEM model (Sohn et al., 2007). From the pattern of ‘Reduc- sis, we suggest that R&D planners need to pay more attention
tion of manufacturing period’, we obtained new information; to the following things in order to increase the performance
if the reduction of manufacturing period increases too much, it of their applicants for R&D funding: whether the candidates of
can hinder productivity, consequently potentially decreasing the R&D funding have a research department or whether their head-
Industrial development and global competitiveness. quarters are located in capital or whether their products are
1542 J.Y. Choi et al. / Research Policy 38 (2009) 1534–1544

positioned in a mature stage of life cycle. However, as we men-
tioned earlier, more specific investigation should be employed to
identify clear reasons for differences in performance. Nonethe-
less, our work can be utilized as a preprocessing task detecting
a suspicious company which may yield poor performance; thus,
this work is also meaningful in an aspect of reducing the cost for
the evaluation process of R&D funding applicants.
The findings from the current study can possibly be utilized by R&D
funding planners or by companies conducting R&D sponsored by
the government to enhance their performances. Nonetheless, this
study has some limitations. The small size of the data set, composed
of 173 company cases, along with the fact that most of the data had
several missing values, contributed to the difficulty of illustrating
a company’s characteristics.
It is necessary to give more specific information to individual
companies because this study analyzed the R&D performance at a
group level, rather than at the level of individual companies. This
is left for the area of further study.
Acknowledgements
This work is based on the Korean report that won the Student
Paper Contest of the Korea National Statistical Office in 2008, which
was supervised by the corresponding author. The current version
of the work was financially supported by the Ministry of Knowl-
edge Economy (MKE) and the Korea Institute for Advancement in
Technology (KIAT) through the Workforce Development Program
in Strategic Technology.
Appendix A.
Tables 8–11.

Table 8
The results of quantification for a set of short- to mid-term performance factors at the second axis.

Factors Measurement variables Dummy Raw coefficient Scores Range Patterns

variables

Technology Improvement of a technological X1 1 0.00 0.38 0.93

performance ability (X1) X1 2 −0.60 −0.22 Negative and positive
X1 3 0.33 0.71
Technological progress (X2) X2 1 0.00 0.15 0.23
X2 2 −0.23 −0.08 Negative and positive
X2 3 −0.16 −0.01
Conquest of a technological gap X3 1 0.00 −0.26 0.47
(X3) X3 2 0.43 0.17 Positive and negative
X3 3 −0.04 −0.30
Localization of the technology (X4) X4 1 0.00 −0.10 0.13
X4 2 0.12 0.02 Positive
X4 3 0.13 0.03

Business performance Improvement of business ability X5 1 0.00 0.11 0.54

(X5) X5 2 −0.10 0.01 Negative
X5 3 −0.54 −0.43
Improvement of a company’s X6 1 0.00 0.10 0.23
popularity (X6) X6 2 −0.09 0.01 Negative
X6 3 −0.23 −0.13
Improvement of market share (X7) X7 1 0.00 −0.60 1.01
X7 2 0.84 0.23 Positive
X7 3 1.01 0.41
Sales and export increase (X8) X8 1 0.00 −0.04 1.30
X8 2 −0.13 −0.17 Negative and positive
X8 3 1.16 1.12

Management Improvement of employment (X9) X9 1 0.00 −0.28 0.91

performance X9 2 0.37 0.09 Positive
X9 3 0.91 0.63
Improvement of R&D environment X10 1 0.00 −0.38 1.21
(X10) X10 2 0.58 0.20 Positive and negative
X10 3 −0.63 −1.01
Improvement of manufacturing X11 1 0.00 −0.14 0.40
environment (X11) X11 2 0.20 0.06 Positive
X11 3 0.40 0.25
Improvement of MIS (X12) X12 1 0.00 0.00 0.39
X12 2 0.02 0.02 Positive and negative
X12 3 −0.37 −0.37

Manufacturing Reduction of product defect (X13) X13 1 0.00 0.01 0.52

performance X13 2 0.03 0.03 Positive and negative
X13 3 −0.49 −0.49
Improvement of productivity (X14) X14 1 0.00 0.28 1.00
X14 2 −0.31 −0.03 Negative
X14 3 −1.00 −0.72
Reduction of manufacturing period X15 1 0.00 −0.67 2.66
(X15) X15 2 0.87 0.20 Positive
X15 3 2.66 1.99
Reduction of manufacturing cost X16 1 0.00 0.12 0.99
(X16) X16 2 −0.33 −0.22 Negative
X16 3 −0.99 −0.87
 J.Y. Choi et al. / Research Policy 38 (2009) 1534–1544 1543

Table 9
The results of quantification for the impact factor at the second axis.

Factors Measurement variables Dummy variables Raw coefficient Scores Range Patterns

Impact Improvement of science and Y1 1 0.00 0.38 0.41

technology (Y1) Y1 2 0.41 0.79 Positive and negative
Y1 3 0.06 0.44
Industrial development (Y2) Y2 1 0.00 0.15 1.59
Y2 2 0.03 0.18 Positive and negative
Y2 3 −1.57 −1.42
Increase of national demand (Y3) Y3 1 0.00 −0.26 2.95
Y3 2 0.89 0.63 Positive
Y3 3 2.95 2.69
Improvement of industrial Y4 1 0.00 −0.10 1.52
structure (Y4) Y4 2 0.82 0.72 Positive
Y4 3 1.52 1.41
Improvement of national economy Y5 1 0.00 0.11 0.54
(Y5) Y5 2 0.26 0.37 Positive and negative
Y5 3 −0.28 −0.17
Improvement of global Y6 1 0.00 0.10 0.85
competitiveness (Y6) Y6 2 −0.33 −0.23 Negative
Y6 3 −0.85 −0.75

Table 10
The results of quantification for a set of short- to mid-term performance factors at the third axis.

Factors Measurement variables Dummy variables Raw coefficient Scores Range Patterns

Technology performance Improvement of a technological X1 1 0.00 −0.49 0.61

ability (X1) X1 2 0.61 0.12 Positive and negative
X1 3 0.37 −0.11
Technological progress (X2) X2 1 0.00 0.42 0.63
X2 2 −0.63 −0.21 Negative and positive
X2 3 −0.45 −0.03
Conquest of a technological gap X3 1 0.00 −0.01 0.24
(X3) X3 2 0.07 0.07 Positive and negative
X3 3 −0.16 −0.17
Localization of the technology (X4) X4 1 0.00 0.25 0.53
X4 2 −0.17 0.08 Negative
X4 3 −0.53 −0.27

Business performance Improvement of business ability X5 1 0.00 −0.16 2.06

(X5) X5 2 −0.05 −0.20 Negative and positive
X5 3 2.01 1.85
Improvement of a company’s X6 1 0.00 −0.72 1.59
popularity (X6) X6 2 0.72 0.00 Positive
X6 3 1.59 0.87
Improvement of market share (X7) X7 1 0.00 −0.13 1.31
X7 2 0.05 −0.08 Positive
X7 3 1.31 1.18
Sales and export increase (X8) X8 1 0.00 −0.22 2.11
X8 2 0.62 0.39 Positive and negative
X8 3 −1.50 −1.72

Management performance Improvement of employment (X9) X9 1 0.00 0.19 0.27

X9 2 −0.27 −0.08 Negative and positive
X9 3 −0.22 −0.03
Improvement of R&D environment X10 1 0.00 −0.20 0.32
(X10) X10 2 0.27 0.07 Positive and negative
X10 3 −0.05 −0.25
Improvement of manufacturing X11 1 0.00 −0.21 1.59
environment (X11) X11 2 0.20 −0.01 Positive
X11 3 1.59 1.38
Improvement of MIS (X12) X12 1 0.00 0.73 1.87
X12 2 −1.01 −0.28 Negative
X12 3 −1.87 −1.14

Manufacturing performance Reduction of product defect (X13) X13 1 0.00 0.35 0.46
X13 2 −0.46 −0.11 Negative and positive
X13 3 −0.04 0.31
Improvement of productivity (X14) X14 1 0.00 0.30 0.39
X14 2 −0.39 −0.09 Negative
X14 3 −0.39 −0.09
Reduction of manufacturing period X15 1 0.00 −0.51 2.51
(X15) X15 2 0.62 0.11 Positive
X15 3 2.51 2.01
Reduction of manufacturing cost X16 1 0.00 0.08 0.31
(X16) X16 2 −0.31 −0.22 Negative and positive
X16 3 −0.16 −0.08
1544 J.Y. Choi et al. / Research Policy 38 (2009) 1534–1544

Table 11
The results of quantification for the impact factor at the third axis.

Factors Measurement variables Dummy variables Raw coefficient Scores Range Patterns

Impact Improvement of science and Y1 1 0.00 −0.49 2.25

technology (Y1) Y1 2 0.02 −0.46 Positive
Y1 3 2.25 1.77
Industrial development (Y2) Y2 1 0.00 0.42 0.76
Y2 2 0.76 1.18 Positive and negative
Y2 3 0.03 0.46
Increase of national demand (Y3) Y3 1 0.00 −0.01 0.53
Y3 2 0.53 0.52 Positive and negative
Y3 3 0.27 0.27
Improvement of industrial Y4 1 0.00 0.25 2.05
structure (Y4) Y4 2 −0.22 0.03 Negative and positive
Y4 3 1.83 2.08
Improvement of national economy Y5 1 0.00 −0.16 0.21
(Y5) Y5 2 0.21 0.05 Positive and negative
Y5 3 0.16 0.00
Improvement of global Y6 1 0.00 −0.72 1.42
competitiveness (Y6) Y6 2 −1.42 −2.14 Negative and positive
Y6 3 −0.44 −1.16

References
Belderbos, R., Carree, M., Lokshin, B., 2004. Cooperative R&D and firm performance.
Research Policy 33 (10), 1477–1492.
Bremser, W.G., Barsky, N.P., 2004. Utilizing the balanced scorecard for R&D perfor-
mance measurement. R&D Management 34 (3), 229–238.
Brown, W.B., Gobeli, D., 1992. Observations on the measurement of R&D productiv-
ity: a case study. IEEE Transactions on Engineering Management 39 (4), 325–331.
Coccia, M., 2001. A basic model for evaluating R&D performance: theory and appli-
cation in Italy. R&D Management 31 (4), 453–464.
Dittrich, R., Francis, B., Hatzinger, R., Katzenbeisser, W., 2007. A paired comparison
approach for the analysis of sets of Likert-scale responses. Statistical Modelling
7 (1), 3–28.
Eder, B.K., Davis, J.M., Bloomfield, P., 1994. An automated classification scheme
designed to better elucidate the dependence of ozone on meteorology. Journal
of Applied Meteorology 33 (10), 1182–1199.
Fisher, R.A., 1946. Statistical Methods for Research Workers, 10th ed. Oliver & Boyd,
London.
Guttman, L., 1941. The Quantification of a Class of Attributes: A Theory and Method
of Scaling Construction. The Prediction of Personal Adjustment. Social Science
Research Council, New York.
Hayashi, C., 1950. On the quantification of qualitative data from the mathematico-
statistical point of view. Annals of the Institute of Statistical Mathematics 2 (1),
35–47.
Hayashi, C., 1952. On the prediction of phenomena from qualitative data on the
quantification of qualitative data from the mathematico-statistical point of view.
Annals of the Institute of Statistical Mathematics 3 (2), 69–98.
Hayashi, C., 1954. Multidimensional quantification—with the applications to analy-
sis of social phenomena. Annals of the Institute of Statistical Mathematics 5 (2),
121–143.
Jonsson, P., Lesshammar, M., 1999. Evaluation and improvement of manufacturing
performance measurement systems—the role of OEE. International Journal of
Operations & Production Management 19 (1), 55–78.
Kaplan, R.S., 1983. Measuring manufacturing performance: a new challenge for
managerial accounting research. The Accounting Review 58 (4), 686–705.
Kerssens-van Drongelen, I., Nixon, B., Pearson, A., 2000. Performance measure-
ment in industrial R&D. International Journal of Management Review 2 (2),
111–143.
Lattin, J., Carroll, J.D., Green, P.E., 2002. Analyzing Multivariate Data. Duxbury.
Oh, D.S., 2002. Technology-based regional development policy: case study of Taedok
Science Town, Taejon Metropolitan City, Korea. Habitat International 26 (2),
213–228.
Pappas, R.A., Remer, D.S., 1985. Measuring R&D productivity. Research Management
28 (3), 15–22.
Revilla, E., Sarkis, J., Modrego, A., 2003. Evaluating performance of public–private
research collaborations: a DEA analysis. Journal of the Operational Research
Society 54, 165–174.
Schumann, P.A., Ransley, D.L., Prestwood, D.C.L., 1995. Measuring R&D performance.
Research Technology Management 38 (3), 45–54.
Small, I., Swann, P., 1993. Business Strategy Review, vol. 4(3). R&D Performance of
UK Companies, pp. 41–51.
Sohn, S.Y., Joo, Y.G., Han, H.K., 2007. Structural equation model for the evaluation of
national funding on R&D project of SMEs in consideration with MBNQA criteria.
Evaluation & Program Planning 30, 10–20.
Suzuki, T., Kudof, A., 1979. Recent application of quantification Japanese medicai
research. Environmental Health Perspectives 32, 131–141.
Tanaka, Y., 1978. Some generalized methods of optimal scaling and their asymptotic
theories: the case of multiple responses–multiple factors. Annals of the Institute
of Statistical Mathematics 30, 329–348.
Tanaka, Y., 1979. Review of the methods of quantification. Environmental Health
Perspectives 32, 113–123.
Venkatraman, N., 1986. Measurement of business performance in strategy research:
a comparison of approaches. Academy of Management Review 11 (4),
801–814.
Werner, B.M., Souder, W.E., 1997. Measuring R&D performance—state of the art.
Research Technology Management 40 (2), 34–42.