J Comput Virol Hack Tech
DOI 10.1007/s11416-017-0305-7
ORIGINAL PAPER
A performance evaluation of information security training
in public sector
Sung-Kyu Park1 · Song-Ha Lee2 · Taek-Young Kim2 · Hyo-Jung Jun2 ·
Tae-Sung Kim2
Received: 3 March 2017 / Accepted: 24 July 2017
© Springer-Verlag France SAS 2017
Abstract Recently, as the incidents of the security breach
and the personal information leakage in public institutions
and the major information/communication infrastructure
have increased, the importance of the development and train-
ing of human resources specialized in cybersecurity, who can
immediately respond to this are emphasized. Accordingly,
the government has announced policies for the development
of human resources, established and operated public sector
cybersecurity training centers; however, there is no method
for understanding the investment performance and effect of
the present cybersecurity education/training in the public sec-
tor. For the establishment of a training system and the quality
control of continuing education, a method for evaluating the
performance of the training is needed, and this can prove the
justification of the promotion of the training program and the
sustainability of the training center. The goal of this study is to
analyze the outcome of education and training in the field of
information security and economic return on investment. For
this purpose, through literature research on the outcome of
B Tae-Sung Kim
kimts@chungbuk.ac.kr
Sung-Kyu Park
parksk0606@gmail.com
Song-Ha Lee
lsh914@naver.com
Taek-Young Kim
ecac5732@naver.com
Hyo-Jung Jun
phdhyo@naver.com
1 Cybersecurity Economics Research Institute, Chungbuk
National University, Cheongju, Republic of Korea
2 Department of Management Information Systems, and
Cybersecurity Economics Research Institute, Chungbuk
National University, Cheongju, Republic of Korea
the domestic and overseas education and training, this study
drew a model that can apply.
Keywords Evaluation training · Cybersecurity · Education
program · Return on investment
1 Introduction
With the development of intelligent and automated vulnera-
bility attack (or analysis) and hacking skill, security breach
incident and personal information leakage increase [9]. In
addition, concern over the occurrence of security risks in the
environments, such as Internet of Things, big data, Cloud
and Mobile increases [1,19]. Thus, as demand for the infor-
mation security professional manpower, who can respond
to them immediately, increase, the importance of education
and training of manpower is emphasized [12]. Especially,
the necessity of the promotion of awareness and education
for the manpower who takes full charge of the information
security of the communication-based facilities of a coun-
try and public institutions is emphasized. In the domestic
public sector, there are problems such as the absence of a
department taking full charge of information security, the
lack of manpower and the limit of the manpower’s exper-
tise while the rate of the implementation of education and
training of each institution is 14.9% as of 2015, which is sig-
nificantly low [11]. Accordingly, the government announced
various policies for the training of manpower specialized in
information security. To realize the manpower development
policy, the government carries out education and training
of manpower in charge of the information security of pub-
lic institutions through educational institutions specialized
in public information security. In promoting the awareness
of information security and improving the capability level,
123

such as knowledge, skill and attitude necessary for business,
the role of the educational institution is important. Educa-
tional institutions need to make efforts such as contacting
lecturers through securing budgets, constructing educational
facilities and infrastructure and developing an educational
curriculum in order to perform the roles in the training and
education of manpower. Since education is operated through
education investment such as certain manpower and bud-
gets by the persons in charge of the education, there is a
need for a method of evaluating the performance accord-
ing to the target of the operation of educational institutions.
Analyzing the effects such as the costs invested in informa-
tion security training, the lecturers’ capability, educational
facilities, educational contents and methods, students’ satis-
faction and how much they apply the contents they learned
through education to their job and understanding the effects
as compared to investments in education can be important
information that can improve the overall process of the oper-
ation of the education in addition to the simple estimation
of monetary values. Moreover, presenting the contribution
decision-making and education concerning the sustainabil-
ity of the information security education carried out in the
public sector to the goals and objectives of the organization
can justify its validity. The goal of this study is to analyze the
outcome of education and training in the field of information
security and economic return on investment. For this purpose,
through literature research on the outcome of the domestic
and overseas education and training, this study drew a model
that can apply.
In Sect. 2 of the paper, we review the related work about
evaluation model of education and training. In Sect. 3, we
propose Phillips ROI methodology’s evaluation process. In
Sect. 4, survey was conducted with the persons in charge
of public institution security, who took the curriculum of
“Strengthening Cyber Security Capability” of the educa-
tional institution and applied to the model.
2 Literature review
An evaluation of the effect of training is process of diagnosis
and evaluating the overall processes of the program, continu-
ously improving the curriculum and proving the justification
of the promotion of the training program and its sustainability
through analysis of management performance because of the
training and the economic effect. This is a process of analyz-
ing the effects, such as the costs invested in the cybersecurity
training, the instructors’ competency, the facilities, contents
and methods of the training, the participant’s satisfaction and
the extent of the application of the contents learned through
the training to the present operations. The evaluation model
related to the effectiveness of the education and training pro-
gram is divided into a process-oriented model that evaluates
123
S. Park et al.
the whole training process, and a result-oriented model that
evaluates the results of training.
2.1 Training process-oriented evaluation model
The IPO model by Bushnell [5] categorized each sub-
domain, considering the characteristics of the education and
training program through the level of evaluations of Element
(Input), Process and Output in the education and training
process and then, presented a model that would carry out an
overall evaluation. Stufflebeam [3] defined the process that
designs, obtains and provides the information necessary for
decision-making through evaluation and proposed a Context-
Input-Process-Performance (CIPP) evaluation model based
on this. The CIPP model is concerned with the content and
character of decision-making and emphasizes the role of the
evaluation responding to that. Lee & Cha [6] applied the
CIPP model to civil servants in local governments, who par-
ticipated in personal information security training. A research
model and hypotheses were set up, which consisted of input,
process, and output. And relationships among them were
analyzed. Effectiveness of education was composed of the
transfer of learning and satisfaction and the interest of the
head of an organization was used as a moderating effect.
The result of the study suggested that the effectiveness of
personal information security education is affected by the
input and process of the education and that the interest
and leadership for education of the head of an organiza-
tion could further strengthen the relationship between them
(input-effectiveness) in a positive direction. Lee et al. [7]
conducted a survey with capital areas civil servants of the
primary local government who had an experience of tak-
ing personal information security education and taking the
reaction (Satisfaction with the education) to the performance
of personal information security education and learning-
behavior-business performance as factors and verified the
hypothesis of the relationship among them, using Structured
Equation Model (SEM). For an evaluation of Context-Input-
Reaction-Outcome (CIRO) model developed by Warr et al.
[16], a model which comprehensively evaluates the context,
reaction and outcome of the introduction of a program was
proposed. The logic model proposed by McLaughan & Jor-
dan [8] includes the factors, including the input, process,
output, outcome and external influence of a program. The
logic model proposed by Wholey consists of two main parts,
the program components and the purpose and effect of the
program. Program logic includes “the logic implied in the
program,” “program components and the achievement of the
goal intended by the program” and “causal relationship con-
nected to the creation of outcome, accordingly”. Grossman &
Salas [14] utilized the transfer process model for an analysis
in order to draw the elements affecting the training transfer,
A performance evaluation of information security training in public sector
such as a positive effect or negative effect when the contents
learned after training is used in a similar environment.
2.2 Training result-based evaluation model
Kirkpatrick [4] composed the model of the outcome of
education and training program with four steps, including
reaction, learning, behavior and result. Reaction means the
satisfaction with an education among the students who par-
ticipated in an education, and the elements of satisfaction
include educational contents and method, education time
and materials, lecturer, educational facilities and environ-
ment, and education operation services. Learning refers to
changes in the capabilities such as knowledge, skill and atti-
tude obtained by the students through the education, which
figures out the degree of the achievement of the goal of aca-
demic achievement set up in advance by the participants in
the education. For evaluations of behavior and result, three
to six months after the completion of the education is sug-
gested as the most appropriate time for the evaluation of the
changes. Behavior evaluation means the practical utilization
of the learned contents in the job, which measures changes
in job behavior according to the education and the transfer
of learning. The result means the evaluation of the positive
impacts of the factors of reaction, learning and behavior
on organizational performance. The U.S. NIST developed
NIST 800-16 “A role-based model for federal informa-
tion technology/cybersecurity training” as guidelines for the
development of human resources for information security in
public sector in 2014. NIST 800-16 described the method of
training needs analysis and evaluation and examples apply-
ing the Kirkpatrick Model [15]. Phillips’s [10] evaluation
model is a model developed by improving Kirkpatrick’s 4-
step model, which focuses on Return on Investment (ROI)
and four-step result evaluation in Kirkpatrick’s model was
divided into the contribution of the education to business
outcome (Business Impact) and ROI. The ROI of education
means the value created by comparing the net profit created
through education with the cost invested in the education.
ROI is the value considering the net profit from education
and training, different from Benefit-Cost Ratio (BCR) con-
sidering the entire benefit from the education and training.
He suggested a model that supplemented the weakness of the
existing models that they could not prove the financial effect
of the education and training program. Kirkpatrick’s and
Phillips’s models are result-based, which conducts an educa-
tion and evaluates its effect, and models that are sequential by
each step, and assume positive correlations. Goh [18] mea-
sured the economic effect of education, conducting a survey
with 162 persons who participated in an e-learning program
to strengthen the job capability of the leaders of the busi-
ness team of a domestic home-study materials company. For
this purpose, Kirkpatrick’s model and Phillips’ model were
analyzed, and a procedural ROI 8-level model was proposed.
Sorensen [17] applied Philips’s model to measure the eco-
nomic effect of the training process of Bosnia & Herzegovina
local governments (Project cycle management curriculum)
and utilized it in data analysis, using questionnaires. As
criteria for analyzing business performance, the number of
project announcements requested for institutions, the num-
ber of projects submitted and the number of projects adopted
and receiving budgets were compared. As major effect factors
for the estimation of ROI, Time Saving and Effect of Project
Implementation were selected for the proposal for a project,
and the ROI result value was estimated through the partic-
ipant’s estimation method. Rafiq [13] applied and utilized
Kirkpatrick’s model with a PIA training center to evaluate
an organization’s training in an analysis. The model used in
the study was applied, predicting that the effect of education
would gradually increase and assumed that a positive effect
would be created as students get support for training from
the organization. Curado & Martins Teixeira [2] improved
and applied Kirkpatrick model in order to draw the training
evaluation level and ROI. The model was utilized, apply-
ing small-scale logistics companies to an analysis by adding
personal factors, environmental factors and motivational ele-
ments to the model. Personal factors include confidence and
the organization’s ability to perform while environmental
factors consist of the organization’s support, learning culture
and business constraint. Motivational elements include atti-
tudes towards career and job, relationships of organizations
and opportunities for training decisions.
3 Research method
The process of the evaluation of the effect of the training
in the cybersecurity sector in this study broadly consists of
four steps. In the step of evaluation planning, the goal of the
evaluation of the training set up, and an evaluation plan is
established. The evaluation plan is to understanding training
needs, using training needs analysis. In the step of data collec-
tion, the method and period of data collection are decided,
according to the training cycle (pre-, mid-, and post-). In
the step of data analysis, the company’s tangible/intangible
performances are analyzed, and the conversion of monetary
value are applied, and finally, the costs put in the curriculum
are calculated to draw ROI value. In the step of reporting is a
step in which improvements of the training are drawn based
on the results of the process evaluation.
3.1 Evaluation planning
The training evaluation process was composed of five levels,
including reaction & satisfaction, academic achievement, job
application, a contribution to performance, ROI estimation
123

Fig. 1 Evaluation process
(Fig. 1) and evaluations were carried out through surveys.
The first survey included Training Needs Analysis, Level 1
Reaction & Satisfaction survey and Level 2 Measurement of
Academic Achievement, and the items on the survey were set
up according to the objectives of the curriculum. The items
of the reaction & satisfaction survey include the relevance of
the curriculum with the job, the importance of the curriculum
in business performance, the application of the curriculum to
the job, the newness of education materials, the level of dif-
ficulty in the curriculum, whether to take the training again
and whether to recommend it to coworkers. The items of
the academic achievement survey were set up as security
awareness and understanding, job-related IT/cyber security
knowledge, specialized skills (Technological and manage-
rial, etc.), tool application and attitude and ability for the
job. Level 3 Job Application is to measure how well they
utilize the contents learned in the training and has the same
items as the Level 2 survey, and three months later, they
were measured through the second survey and compared with
the results of the first survey. The items in the survey were
used based on a seven-point Likert scale. In Level 4, for
contribution to performance, business performance through
education was divided into individual performances and busi-
ness performances. The individual performances were set
up as the improvement of job satisfaction, the decrease of
job stress, the improvement of teamwork, the increase of the
contribution to the organization and the improvement of com-
munication ability. The business performances were set up
and measured by the decrease of withdrawal of manpower,
the improvement of productivity and the decrease of the risk
level (Table 1).
3.2 Data collection
This study conducted an evaluation with the holders of a
position in the organization taking full charge of public insti-
123
S. Park et al.
tutions, who participated in the education for strengthening
cyber security capability. From June 13 through September
23, 2016 when the curriculum was in progress, the first survey
was conducted at the time of the completion of the curricu-
lum. The second survey was conducted three months later on
the date of the completion of the curriculum in October 2016.
The first survey (Reaction & satisfaction, academic achieve-
ment) distributed questionnaires via e-mail, and 119 copies
in total were collected, and the second survey was conducted
three months later. Questionnaires were distributed to 57 per-
sons, and 13 copies in total were collected for the use in an
analysis.
3.3 Data analysis
At the level of data analysis, to isolate the effect of educa-
tion on business performance measured through the second
survey, the participant’s estimation method was used. The
participant’s estimation method is a method of obtaining
direct information from the students who had an education,
and it applies under the premise that the students who par-
ticipated in the education themselves may have an ability to
estimate or determine the change of business performance
through the education with a higher reliability than persons
outside, who did not. To convert the isolated educational
performance to a monetary value, the benefit was estimated
through the productivity improvement indicator of the busi-
ness performance indicators and the costs invested in the
education program were drawn through an interview with
operators of educational institutions.
ROI calculation is Equation (1).
T otal Pr ogram Bene f it − T otal Pr ogram Cost
ROI = ×100
T otal Pr ogram Cost
(1)
A performance evaluation of information security training in public sector
Table 1 Training evaluation
Level Phillips ROI method
process (Phillips ROI method
versus Kirkpatrick model) 1 Reaction & planned action
2 Learning
3 Job application
4 Business impact
5 Return on investment
Table 2 Reaction & planned action data
No Measures
1 Importance of educational contents in business
2 Up-to-dateness of educational contents
3 Relevance of education materials for business
4 Level of difficulty in educational contents
5 Willingness to take again
6 Willingness to recommend
Average
4 Evaluation: an illustrative example
4.1 Reaction & planned action (level 1)
As a result of measurement of items in Lv1 evaluation on
a seven-point scale, the overall satisfaction was high (5.41
points on average). Especially, willingness to take the cur-
riculum again (6.16) and willingness to recommend (6.00)
were the highest. On the other hand, it was found that most
of them were unsatisfied with the level of difficulty in edu-
cational contents (4.01) (Table 2).
4.2 Learning & job application (level 2 & 3)
Academic achievement (Learning) was 5.43 on average, and
security awareness and understanding were the highest (6.0).
In contrast, achievement of tool application (Software and
hardware, etc.) was the lowest (4.71). For the job applica-
tion measured three months later after the completion of
education, security awareness and understanding were the
highest (6.0). For the job application, too, tool application
was the lowest (4.2). For the job knowledge learned through
the curriculum, job application improved by 0.1 while for
the remaining 4 items, job application was low. For the job
application facilitators, the suitability of learning contents
with the job and the usability and newness of learning con-
Description Kirkpatrick model
Measures participant satisfaction with the Reaction
program and captures planned action
Measures changes in knowledge, skills, Learning
and attitudes
Measures changes in on-the-job behavior Behavior
and progress with application
Captures changes in business impact Results
measures
Compares program monetary benefits to
the program costs
tents were in the first place (38.5%), and for learning barriers,
Score enough time and environment for job application were in the
first place (33.4%) (Tables 3, 4).
5.55
5.51 4.3 Business impact (level 4)
5.26
4.01 For individual performance through education, most respon-
6.16 dents responded that job satisfaction improved (5.7 points),
6.00 followed by the increase of the contribution to the orga-
5.41 nization (5.4). For the organization performance measured
through the participants’ estimation method, the respon-
dents responded that the improvement of productivity had
the biggest impact on the organizational performance, and
they responded that the education contributed about 55.4%
to the improvement of productivity. Along with this, they
responded that education made a contribution of about
54% to the reduction of risk level, another organization
performance, too, but for the decrease of withdrawal of
manpower, the education’s contribution was 39.2%, which
was somewhat low. The qualitative educational performance
suggested by the students included the promotion of infor-
mation security awareness, the improvement of individual
business ability and the improvement of the organization’s
security level. The average time for which the respondents
performed information security business was 3.7 hours,
and the time of the information security business reduced
after the education was 7.9 hours a month on average (per
day). For the desirable hours of education and training,
they wanted 45 hours a year (about 6 days) and 1.6 mil-
lion won a year as an educational support fund (Tables 5,
6).
4.4 Return on investment (ROI, level 5)
Before the estimation of monetary benefits according to edu-
cation, the effect of education on organizational performance
indicators was isolated, using the participants’ estimation
method. It was found that the indicator with the highest
effect of education on organizational performance indica-
123
 S. Park et al.

Table 3 Learning & job

No Measures Learning (after the com- Job application Gap
application data
pletion of education) (3 month later)

1 Security awareness/understanding 6.0 6.0 0.0

2 Job knowledge 5.7 5.8 +0.1
3 Specialized skills 5.13 5.1 −0.03
4 Tool application (software/hardware) 4.71 4.2 −0.51
5 Job attitude 5.65 5.6 −0.05
Average 5.43 5.34 −0.09

Table 4 Factors affecting job

No Factors (support/lack) Facilitators (%) Barriers (%)
application (facilitators and
barriers) 1 Senior officer’s continuous feedback 15.3 0
2 The management’s interest, support and backing 7.7 33.4
3 Usability and newness of learning contents 38.5 0
4 Suitability of learning contents for the job 38.5 8.3
5 Sufficient time and environment for job application 0 50
6 Other (no barriers, etc.) 0 8.3

Table 5 Individual performance according to security training (intan- Total Monetary Beneit = 10,855,938 Total Cost = 1,193,634
gible)
−
No Individual performance Score ROI = × 100
1 Improvement of job satisfaction 5.7 10,855,938−1,193,634
= × 100 = 809%
2 Decrease of job stress 5.0 1,193,634
3 Improvement of teamwork 4.5
4 Increase of contribution to the organization 5.4
5 Improvement of communication ability 5.1 Fig. 2 ROI value
Average 5.14

education program consist of the lecturer cost, educational

tors was the “job productivity” indicator (20.6%). This study
analyzed the educational monetary benefit of the job produc-
tivity indicator. As a result of the analysis, monetary benefit
of 10,855,938 won was produced in total, and the mone-
tary benefit from the improvement of job productivity was
835,072 won on average per person. The costs invested in the
preparation cost, textbook publication cost, meeting cost and
reserve fund. As a result of an interview with the operators
of educational institutions, the cost per person was 91,818
won. When that was converted to 13 persons, the cost was
1,193,634 won. When the curriculum was operated with 13
persons, the ROI was 809% (Fig. 2, Table 7).

Table 6 Organization performance according to security training

No Organization performance Indicator 1. decrease of Indicator 2. improvement of Indicator 3. decrease of
indicators withdrawal of manpower job productivity (%) risk level (%)
(%)

1 Contribution of indicator improvement to 43.1 51.9 50.4

organization performance
2 Educational contribution to indicator 39.2 55.4 54.0
change
3 Reliability of the estimated contribution 61.9 71.9 65.0
Isolated effect of education 10.4 20.6 17.6
Effect of education = organization performance change × education contribution × reliability of estimation

123
A performance evaluation of information security training in public sector

Table 7 Estimation of monetary benefit of education

No Reduced business Annual personnel Change of organization performance Participants’ Reliability of the Monetary
hours expenses (reduced personnel expenses) estimation (%) participant’s esti- benefit
mation (%)

1 15 42,500 3,988,375 40 40 637,500

2 0 65,600 0 30 95 0
3 2 65,000 812,500 70 70 398,125
4 3 27,500 515,625 40 80 165,000
5 0 27,500 0 90 90 0
6 2 42,500 531,250 0 100 0
7 40 42,500 10,625,000 50 60 3,187,500
8 16 65,000 6,500,000 80 80 4,160,000
9 5 27,500 859,375 80 20 137,500
10 10 27,500 1,718,750 70 70 842,188
11 10 42,500 2,656,250 50 100 1,328,125
12 0 42,500 0 40 40 0
13 0 65,000 0 80 90 0
Total 10,855,938

5 Conclusions general performance indicator that came under all training.

This study proposed a research method that applies a model
for the evaluation of the effect of information security train-
ing and carried out evaluations with the holders of a position
in the organization taking full charge of security in public
institutions through surveys. The educational performance
model aims to diagnose the overall training process and
draw improvements, instead of simply estimating micro-
scopic ROI in training. Accordingly, by diagnosing each
level of educational satisfaction, academic achievement, job
application, business performance and ROI, improvements
in the operation of the curriculum can be drawn based on the
result of an analysis. In addition, in order to understand the
application of knowledge, skill and ability learned through
an education to the job, educational institutions should carry
out the follow-up management of students. When the model
proposed in the study applies, an evaluation of systematic
education and training would help improve the level of pub-
lic sector information security manpower and promote the
national information security competitiveness. In this study,
we investigated the effect of information security training
support on business performance. In order to grasp the growth
rate of business performance, we have found the rate of
change of ‘job productivity’ indicator and used it in an anal-
ysis of the ROI in training. The key factor in the evaluation of
the effectiveness of training is not simply to derive the score
(ROI), but to ‘improve the problem by checking the overall
process of education’. Based on the results of this study, we
aim to contribute to the training of high-skilled information
security personnel in the public sector by improving the qual-
ity of training. The limitation of this study is that it used a
In the future, it would be necessary to conduct additional
research in order to draw key performance indicator of the
information security organization of public institutions and
to understand the factors of training affecting the business
performance of public institutions.
Acknowledgements This research was supported by the Basic Sci-
ence Research Program through the National Research Foundation
of Korea funded by the Korean Government (NRF-2011-0025512).
This work was supported by the Ministry of Education of the Repub-
lic of Korea and the National Research Foundation of Korea (NRF-
2015S1A5A2A01009763). This study was carried out with the support
of ‘R&D Program for Forestry Technology (Project No. 2016004C10-
1619-AB01)’ provided by Korea Forest Service.
References
1. Lee, A.: Authentication scheme for smart learning system in the
cloud computing environment. J. Comput. Virol. Hacking Tech.
11(3), 149–155 (2015)
2. Curado, C., Martins Teixeira, S.: Training evaluation levels and
ROI: the case of small logistics company. Eur. J. Train. Dev. 38(9),
845–870 (2014)
3. Stufflebeam, D.L.: Toward a Science of Educational Evaluation.
Educational Technology Publication, Englewood Cliffs (1973)
4. Kirkpatrick, D., Kirkpatrick, J.: Evaluating Training Program: The
Four Levels, 3rd edn. Berrett-Koehler Publishers, San Francisco
(2006)
5. Bushnell, D.S.: Input, process, output: a model for evaluating train-
ing. Train. Dev. J. 44(3), 41–44 (1990)
6. Lee, H.J., Cha, Y.J.: A study on the effectiveness of privacy edu-
cation using the CIPP model: focusing on the perceptions of local
government officials. Korean J. Local Gov. Stud. 19(1), 95–119
(2015)

123

7. Lee, H.J., Lee, H.C., Cha, Y.J.: A study on the performance of pri-
vacy education in local government: focused on Kirkpatrick model.
Korean J. Local Gov. Stud. 24(1), 173–196 (2015)
8. McLaughlin, J.A., Jordan, G.B.: Using logic models. In: Handbook
of practical program evaluation, pp. 7–32. Jossey-Bass Inc., San
Francisco (2004)
9. Kim, J.H., Lee, K.H.: Vulnerabilities of intelligent automobiles
using TPEG update based on T-DMB and its countermeasures. J.
Comput. Virol. Hacking Tech. 11(3), 143–148 (2015)
10. Phillips, J.J.: Handbook of Training Evaluation and Measurement
Methods. Routledge, New York (1997)
11. Korea Internet & Security Agency: National information security
white paper. Ministry of Science, ICT and Future Planning (2016)
12. Choi, M.K., Han, K.S.: A study on the service quality of SMEs
management consulting affecting the perceived management per-
formance in mobile communication environments. J. Comput.
Virol. Hacking Tech. 10(2), 145–156 (2014)
13. Rafiq, M.: Training evaluation in an organization using Kirkpatrick
model: a case study of PIA. J. Entrep. Organ. Manag. 4, 151 (2015)
123
S. Park et al.
14. Grossman, R., Salas, E.: The transfer of training: what really mat-
ters. Int. J. Train. Dev. 15(2), 103–120 (2011)
15. Toth, P., Klein, P.: A Role-Based Model for Federal Informa-
tion Technology/Cybersecurity Training. NIST Special Publication
800-16, Gaithersburg (2014)
16. Warr, P., Bird, M., Rackham, N.: Evaluation of Management Train-
ing. Gower Press, London (1970)
17. Sorensen, S.E.: Comprehensive return on investment evalua-
tion system for local self-government training in Bosnia and
Herzegovina—case: project cycle management course. Perform.
Improv. 54(1), 14–27 (2015)
18. Goh, Y.S.: Design of the procedural ROI process model for human
resource programs. Korean Soc. Educ. Technol. 21(2), 153–186
(2005)
19. Kim, Y.T., Jeong, Y.S.: A token-based authentication security
scheme for Hadoop distributed file system using elliptic curve cryp-
tography. J. Comput. Virol. Hacking Tech. 11(3), 137–142 (2015)