You are on page 1of 19

Developing Key Performance Indicators

for the R&D Programs using Logic Model


and Matrix Map
Ji hyun Park, Researcher (jihyun@kistep.re.kr)
Sang ki Jeong, Director (sjeong@kistep.re.kr)
Dong hoon Oh, Research Fellow (smile@kistep.re.kr)
November 3 2006
R&D Industry & Infrastructure Evaluation Team of R&D Evaluation Center
Korea Institute of Science and Technology Evaluation and Planning (KISTEP)

Contents
I

Background for developing Standard Performance Indicators for the


evaluation of R&D programs
- Status of R&D program evaluation in Korea
- Necessity of performance indicators

II

Methodologies for developing Performance indicators and Key


Performance Indicators reflecting R&D characteristics
- Case studies for the development of performance
indicators using Logic Models
- Setting up Key Performance indicators on the Matrix

III

Challenges

History of Research & Development program evaluation

2006

Performance based evaluation of R&D programs was


implemented with Act on performance evaluation and
management of national R&D program.

2005

Program evaluation was output oriented.


It was a monitoring system to evaluate R&D programs
in a short term every year.

1999

Program evaluation was started by NSTC*


* NSTC : National Science and Technology Council

Status of national R&D evaluation in Korea

Policy/Planning

NSTC
(National Science and
Technology Council)

Social & economic


effect

R&D Program

Ministries/
Research Councils

Program
Outcome & Impact

Project

R&D
Centers
Funded
By Gov.
(R1)

Project
(R2)

Project
(R3)

Project
Technical Output

Evaluation system of national R&D programs

Performance based evaluation of R&D programs

Specific Evaluation

Meta Evaluation

Self Evaluation

Periods

Feb ~ July

May

Feb ~ Apr

Subject

NSTC

NSTC

Ministries

Summary

In-depth
evaluation issue
oriented and
comparative
analysis

Evaluation for the


results of self
evaluation

Evaluation for the


achievement of
program
objectives using
performance
indicators

Overview of the national R&D performance measure system


National
National policy
policy level
level
Future
Future
image
image

Policy
Policy goal
goal

Strategic
Strategic plans
plans

Key
Key strategies
strategies
(National
(National plans
plans of
of each
each areas)
areas)

Generic
Generic LM
LM

Standard
Standard PI
PI
(SPI)
(SPI)

Develop
Develop logic
logic models
models

goals
goals

Distribute
Distribute key
key strategies
strategies

Vision
Vision

Individual
Individual project
project Level
Level
Private
Private projects
projects

Main
Main projects
projects

Performance indicators at the projects levels

Ministrial
Ministrial program
program level
level
Program
Program PI
PI

Key
Key programs
programs

Mutual
Agreement

Program
Program planning
planning

MOST
aa

Progrms
Progrms
AA

BB

......

bb

...............

.
.

cc

100%
100% 50%
50% %
%
.
.

Initiatives
Initiatives

Standard national R&D program classifications


Guideline for the self-evaluation provides Standard Performance
Indicators with R&D program classification.

Program objectives

Research & Development


Development of Human Resources
International Cooperation
Utility/Equipment. Etc.

Result utilization
Basic Knowledge
- Pure basic, Specific basic
Industrial Technology
- Short term, Mid-Long term
Public Services/Region/etc.
- Public service, BioAgriculEnvir, Region

National master plans of S&T in Korea


Periods

Involved ministries

Area

Improving Basic
research

2006~2010

Ministry of Science &Technology,


Ministry of Education & Human
Resources Development

Basic research

Innovation of
industrial
technology

2004~2008

Ministry of Commerce, Industry


and Energy

2005~2007

Ministry of Science &Technology,


Ministry of Commerce, Industry
and Energy etc.

Region

2000~2006

Ministry of Agriculture & Forestry,


Ministry of Maritime affairs &
fisheries, Forest government
Information Agency etc.

Agriculture &
Forestry

Fostering
Bioscience

2002~2006

Ministry of Health & Welfare,


Ministry of Science &Technology,
etc.

BioHealth

Development of
Environmental
technology

2003~2007

Ministry of Environment, Ministry


of Science &Technology etc.

Environment

Plans

Improving regional
science &
technology
improving
Agriculture &
Forestry

Industrial
technology

Example : Developing standard logic model from the national plans


(Bio area)
Back
ground

Input

Activity

Output

Increasing
concerns on
bioscience

Securing genetic
resources and
promoting utilization

Human resource Facilities


Gov.
Private
and
in Bioscience
investment
investment
information
-5,1620 billion -7,7455 billion -9392(00yr)
system

Weak industryacademic cooperation


and information
network

Weak
competitiveness
in technology

Related institution and acts

Real time analysis


and monitoring on
invest portfolio

-Biotechnology promotion law


-Biotechnology promotion plan

Overall Coordination

R&D

Industrialization

Infrastructure

Arranging
R&D
priority

setting and program

system
coordination

Expanding R&D
areas and
concentrating on
selected areas

Fostering
intensively BT
industry

Strengthen Supplying Global


System
ing HRD research networking maintenance
in BT
facilities

Coordination
& allocation

Expanding R&D
in New tech and
fusion tech area

Fostering Promoting
venture, industrializ
small co.
ation

Establishing
policies

Increase of
government
investment

Out
come

Increasing
interests in bio
safety

Upgrade of policy
analysis and
evaluation system

Concentrating on
new tech and mixed
tech development

Creating high values


and enforcing bio
resources

Concentrating on selected BT
areas

Develop Construction Construct


-ion of
ing HR of research internatio
centers
&
in BT
-nal
systems
linkages
area

Strengthening Policy for


cooperation industriali
between co.
zation

Strong Active
specialty network

Fostering industries
strategically

Intensively fostering Bio


industry as a strategic area

Constructi
-on of
systematic
infra

Building EstablishR&D hubs ment of


for global systematic
networking infra

Strengthen public and


R&D infrastructure

External
factors

Policy
Issues

Government

Policies

Uncertainty
of R&D

Social
interests

Example : Performance indicators from the logic model (Bio area)

Back
ground

Input

Activity

Output

Social cognition
about importance
of Bio-tech

Promoting
application of
future genetic
resources

Human resources involved


Gov/private investment
Portion of investment of BT in BT research area
in a whole R&D investment Ratio of high level HR

Research
facilities and
information
systems

Status of industryacademic cooperation and


information network

International
competitiveness

Related system and laws

Ratio of R&D budget


depending on various
standards

-Biotechnology promotion law


-Biotechnology promotion plan

Overall Coordination

R&D

Industrialization

Infrastructure

Arranging
R&D
priority

setting and program

coordination
system

Expanding R&D
areas and
concentrating on
selected areas

Fostering
intensively BT
industry

Strengthen Supplying Global


System
ing HRD research networking maintenance
in BT
facilities

Fostering ventures and


small co.
Utilization and
commercialization of BT

Development of HR and research institutions in BT area


Construction of research centers/bio-complex
Building R&D hubs for international cooperation and attracting
international co.
Protection of system maintenance such as national safety, bio
ethics and intellectual property rights
Application of bio resources

Minimizing redundancy and


duplication
Establishing related policies

Rates of government
investment

Out
come

Public concerns about


national security and
bioethics etc. related
with Bio tech

Analyzing in a view of
industrial economy
Utilizing systems such as
evaluation system

Development high, new


and fusion technologies
(papers, patent etc.)

Upgrade of R&D capacity


Diffusion of research results
(Public relations etc.)

Market share by creating


high values

Internal bio tech degree comparing with


developed countries
Advanced bio technology

Revitalization of tech transactions


business to business
Success rates of utilization and
commercialization

Extending the share of the


market (Effect of industry
development)
Improving productivity
(Effect of reducing costs)

Improvement of specialties in BT area


Utilization of network such as information & facilities
Construction of international cooperation system
Standardization and internationalization of BT
Preservation of bio resources for the national benefits

Informing about Bio tech to people


Establishing c-GMP and bioplants
Strengthening international basis for clinical study
Certifying safety of LMOs

Ratio of BT industry among the main


strategic industries
Important strategic tech in BT industry

External
factors

Policy
Issues

Government

Policies

Uncertainty
of R&D

Social
interests

Setting up Key Performance Indicators on the Matrix

Positioning R&D programs in a


group of area on the Matrix

Setting up key performance indicators


of each area and utilizing other areas
indicators as indirect performance
indicators

Need for analysis of specific characteristics in different areas


Area
Environment

Construction &
Transportation

Health care
Aerospace &
astronomy
Information &
Communication
Machinery &
Materials
Energy
Agriculture &
Forestry

Technological & Industrial Characteristics


-Too strong on public goods
-Influenced on the law, system and restriction
-A technology intensive industry
-Government lead mass programs
-Strongly influenced by government on decision making such as construction of new
town and highway
-Too concentrated on centralization of capital
-Many beneficiaries should be satisfied in a short time
-Public area needs for research on aging society
-Ongoing information industrialization is necessary to open health care information
-Too concentrated on public & basic research
-Benefits are invisible in spite of investment
-Long-term planning is necessary
-Development of industry is very fast
-Progressed rapidly fusion interlaboratory
-Nongorvernment lead area
-Basic industrial technology
-A technology intensive industry in various area
-Outputs are produced in short term
-Influenced on international restriction such as national energy safety & the Climatic
Change Convention
-Too strong on public research
-No key trends due to various area involved
-low level infrastructure

Example : Key Performance Indicators

Performance indicators of BioHealth area

Key
Performance
Indicators

Indirect
Performance
Indicators

Technology (know-how) transfer


Strengthening health care technology
Increasing competitiveness of health industries
Information industrialization
Improve customer satisfaction for public service

Papers
Patents
Effect on making social & economic profits
Advances in technology development
Public based technology
Reinvigorate local economy
Productivity increase
Development of human resources in biohealth area
International cooperative research
Construction and utilization of research facilities

Process of developing R&D program performance indicators

Logic model from R&D


Policy & National
master plan

Standard
Standard
Performance
Performance
Indicators
Indicators

Analysis of technical &


industrial
characteristics of R&D
area

Key
Key
Performance
Performance
Indicators
Indicators

Setting
Settingup
up
effective
effective
performance
performance
indicators
indicatorsofofeach
each
R&D
R&Dprogram
program

Logic model of
individual program

Introduction of R&D commercialization for Genome program


Objectives :
- Establish infrastructure for national resources and high tech medical technology
- Promote advanced clinical technology through disease research
- Realize genomic medicine by the predictive and tailored medicine

Period : 20012010
Budget : 1,220 billion
Flow of activities
1st (01~03)

2nd (04~07)

3th (08~10)

4th(11~15)

Establishment of
infrastructure for
genomic research

Genetic analysis
related life and
disease

Commercializat
ion of genomic
research

Simulation of
commercialized
techniques

Program contents
- Planning to analyze Korean genome
- Genome based R&D commercialization
- Development of applied and predictive technology
- Analysis of genes which are related with life cycle and diseases

Logic model of R&D commercialization for Genome program

Background

Input

Activity

Output

Outcome
Infra

Problems
related
aging
society
Increasing
average
span of
humans
life
Accomplis
hment of
Genome
projects
Diseases

National
resources
Government
invest
1,220billion
National
long-term
plans
(Genetic
research
fostering
plan)

Genomics

Disease
research

Preservation of
bio resources
Construction of
information
Research on
gene function

Results of
disease
research

Development
of high tech
medical
technology

Development
of High /
Predictive
tech

Genomic
medicine
(Predictive,
tailored
medicine)

Healthcare
/disease
manageme
nt program

R&D

Stabilization
UtilizationStabilization
Utilization

Development of
resource genetic
information
Establishment of
base tech
system

Forming
cooperation
networks
Accomplishin
g genetic
information
delivery
Developing
disease related
genetic markers
Analyzing
genes
generating
disease from
animals

Systematiz
ation of
disease
knowledge
Utilization of
predictive tech
Construction of
cooperation for
diagnosis and
medical
treatment

Realization of
Development of utilizing tech
public health
Protection
policy/program

disease in
advance

Selecting important indicators among the performance indicators generated from program logic model and
deciding final program KPI by mutual agreement between the NSTC and program managers.

Performance indicators of Genome based R&D commercialization program

Activity

Output

Outcome

Development of
high tech
medical
technology

Genomic medicine
(Predictive, tailored
medicine)

Genomics

Disease
research

No. of
developing
genomic
characters

Results of
research on
gene functionpapers,
patents

Results of
research on
diseasespapers,
patents

Results of
developing
high/predictive
tech papers,
patents

Rates of
guaranteein
g bio
resources

Construction of
international
cooperation
network

Development of
gene markers
related specific
disease

Rates of
genetic
information
DB

Analysis of
Korea-specific
and target
diseases genetic
characteristics

Mapping of
specific
diseases related
genes

Utilization of
high/predictive
tech
-Developing
program for
disease
prediction
-Covering
information
system for direct
response

Degree of
advance in
public health
policy/program
comparing with
developed
countries

Rates of
cooperating for
developing
diagnosis and
medical treatment

Rates of early
diagnosis of
diseases

National
resources

No. of
constructing
information DB

Rates of
developing
human
resources
-Specialized
HR
-Teaching
materials

Accomplishment
of genetic
information
delivery map

Degree of
establishing
basis for
genetic
information
system

Heath care
program
management

Degree of
utilizing
medical tech

Challenges

Logic Model was used to develop performance indicators of R&D


programs.

Performance indicators of R&D program should be developed through


mutual agreement between government and each ministry to decide
whether programs performance indicators reflect the national master plan.

R&D programs were evaluated output oriented but now on it would be


evaluated focusing on the performance such as outcome and impact. So,
further research would be developing methodologies how to measure the
impact of R&D programs.

Peer review and survey are necessary to verify the key performance
indicators of specific area and to decide allocation of them.

Thank you !
Presented by Ji Hyun Park
Homepage| www.kistep.re.kr
E-Mail| jihyun@kistep.re.kr
Tel| +82+82-02-589589-2275