See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/275037404

Evaluating and Monitoring the Socio-Economic Impact of Investment in

Research Infrastructures

Technical Report · January 2015

DOI: 10.13140/RG.2.1.2406.3525/1

CITATIONS READS

3 1,379

3 authors:

Alasdair Reid Elina Griniece

European Future Innovation System Centre European Future Innovation System Centre
107 PUBLICATIONS   536 CITATIONS    17 PUBLICATIONS   23 CITATIONS   

SEE PROFILE SEE PROFILE

Jelena Angelis
European Future Innovation System Centre
12 PUBLICATIONS   13 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

An assessment of the cumulative cost impact of specified EU legislation and policies on the EU forest-based industries View project

CCA of the Forest Based Industries - DG GROW European Commission View project

All content following this page was uploaded by Alasdair Reid on 17 April 2015.

The user has requested enhancement of the downloaded file.

2015

Evaluating and Monitoring the Socio-

Economic Impact of Investment in
Research Infrastructures

www.technopolis-group.com
Evaluating and Monitoring the Socio-
Economic Impact of Investment in Research
Infrastructures

technopolis |group|, 2015

Elina Griniece, Alasdair Reid and Jelena Angelis

www.technopolis-group.com
Table of Contents
1. Introduction 1  
1.1 Why evaluate the socio-economic impact of research infrastructures? 1  
1.2 Previous evaluations of the impact of research infrastructure investment 2  
1.3 The aim of this guidance document 3  
2. Logical framework for socio-economic impacts of investment in research
infrastructure 4  
2.1 Design and construction phase 4  
2.2 Operational phase 7  
3. Impact assessment methods and tools 13  
3.1 Impacts on economy 13  
3.2 Impacts on human resource capacity 14  
3.3 Impacts on innovation 14  
3.4 Impacts on scientific activity 15  
3.5 Impacts on society 15  
3.6 IA implementation process, its challenges and limitations 17  
Bibliography 18  

Table of Figures
Figure 1: Typology of research infrastructures ................................................................ 1  
Figure 2: Stakeholders and beneficiaries of investment in research infrastructure ....... 2  
Figure 3: Logical framework for socio-economic impact assessment of investment in
research infrastructure ..................................................................................................... 5  
Figure 4: Indicators for economic impacts of RI construction phase ............................. 6  
Figure 5: Indicators of innovation impact of RI construction phase .............................. 7  
Figure 6: Indicators for economic impact of RI operational phase ................................. 8  
Figure 7: Indicators for impact of RI operation on human resource capacity ................ 9  
Figure 8: Indicators for impacts on innovation of RI operational phase ...................... 10  
Figure 9: Indicators for impact on scientific activity ..................................................... 12  
Figure 10: Indicators for impacts on society .................................................................. 13  
Figure 11: Implementation of an impact assessment of research infrastructure .......... 16  
1. Introduction
1.1 Why evaluate the socio-economic impact of research infrastructures?
Research infrastructures (RI) refer to facilities, resources (including human) and
related services needed by the research community to conduct research in any
scientific or technological field. Research infrastructures include:
• Major equipment or group(s) of instruments used for research purposes;
• Permanently attached instruments, managed by the facility operator for the
benefit of researchers, industrial partners and society in general;
• Knowledge-based resources such as collections, archives, structured
information or systems related to data management, used in scientific
research;
• Enabling information and communication technology-based (ICT) or ‘e-
infrastructures’ such as grid, computing, and software communications;
• Any other entity of a unique nature that is used for scientific research.1
Due to the large number of research communities and complex research needs, there
are very different types of research infrastructures with specific characteristics. Figure
1 provides an overarching typology of RIs.
Figure 1: Typology of research infrastructures
Type of research
Description Examples
infrastructure
Unified body of High-performance laser system,
Single-site facility equipment at one physical clean room, coastal observatory,
location centre of competence
Network of distributed ELI: European Light
instrumentation or Infrastructure; Council of
Distributed facility
collections, archives and European Social Science Data
scientific libraries Archives
Mobile vehicles specially
Research vessels, satellite and
Mobile facility designed for scientific
aircraft observation facilities
research
ICT-based system for
European Grid Computing
scientific research,
Infrastructure; Digital Research
Virtual facility including high-capacity
Infrastructure for the Arts and
communication networks
Humanities (DARIAH)
and computing facilities

Setting-up, or renovating, RI usually requires a considerable level of financial

investment and a long-term operation strategy. For example, in the environmental
sciences, many research infrastructures bring their scientific return only after decades
of sequential data recording. The investment strategy requires careful planning of the
operation phase and possible future reinvestments. This entails the purchase of
technologically advanced equipment, clustering of specific skills to enable its
deployment and devising appropriate governance structures. RIs can also be closely
linked to other research and innovation establishments, such as vocational schools,
universities, private and public research centres, research hospitals, business
incubators and science and technology parks.

1 Definition used by the European Commission

Evaluating the socio-economic impact of research infrastructures

© Technopolis Group, 2015 1
While RIs are designed for research needs, the impacts of these facilities reach beyond
fuelling scientific excellence. The advanced technical opportunities and the
concentration of skilled human capital and know-how can foster innovation, create
new or expand the existing markets, attract inward investment, increase economic
activity and potentially have an impact on the social and cultural life in a particular
region. In this regard RIs can be viewed as focal points for continuous interaction
between scientific, technological and socio-economic development.2
The construction and operational phase of RIs are largely funded from public budgets
either through national funding from one or several countries, or a mix of national and
EU funding. For this reason it is crucial to understand the return on investment in RI
to support informed decision-making. Yet it is difficult to quantify and understand this
return in conventional commercial terms. Investment in RI brings a broad range of
benefits that spreads across wider society rather than serving merely the direct
stakeholders (owners and users of RI). Official statistics do not sufficiently describe
the variety of benefits associated with the development and, more importantly,
exploitation of RI. It calls for more elaborate and fine-tuned approaches to account for
the impacts that the RI investment brings on science, economy and society.
Figure 2: Stakeholders and beneficiaries of investment in research infrastructure

1.2 Previous evaluations of the impact of research infrastructure investment

Currently there is no unified framework for the impact assessment (IA) of investment
in research infrastructures. Various conceptual frameworks exist in parallel
comprising a range of observable direct and indirect effects and longer-term impacts.3
The existing studies focus on three types of impacts: 1) the direct and indirect
economic benefits of spending large amounts of public money in a single location; 2)
the industrial knowledge spillovers realised by contractors that design, build and
equip research facilities and spin-offs that provide specialist technical services back to
the facility (or other facilities); and 3) the local economic effects and high-technology

2 Rizzuto, C. (2012) Benefits of Research Infrastructures beyond Science, presentation at ERF Workshop
“The Socio-Economic Relevance of Research Infrastructures”, 31 May-1 June 2012, Hamburg
3 See, for example, project Research Infrastructures: Foresight and Impact (RIFI), project on Evaluation of

Research Infrastructures in Open innovation and research systems (EVARIO), Research Infrastructure
Group of UK Science & Technology Facilities Council, Czech Metodika: Evaluation Methodology for
Research Infrastructures

Evaluating the socio-economic impact of research infrastructures

© Technopolis Group, 2015 2
clusters that grow up around some of the larger facilities.4 A heterogenous set of
methods is applied to capture these effects of RI. Most of them address standard
economic impacts (direct effects) and to some extent economic multipliers.5
Comprehensive and methodologically demanding studies are still rare. Core aspects of
RI benefits, such as their impact on human and social capital formation and
innovation, are not extensively explored. Up to date, the existing literature provides
insufficient evidence to support claims that investment in RI (even large-scale) attract
and retain talent and promote innovation.6 Therefore, important socio-economic
contributions remain poorly understood. In this respect, significant work is
undertaken by FP7 project “Research Infrastructures: Foresight and Impact (RIFI)”. A
Foresight Enriched Research Infrastructure Impact Assessment Methodology
(FenRIAM)7 was developed during the project, which significantly contributed to the
understanding of the impact of RI on learning and capacity of RI operators, suppliers
and users. A key lesson is that there is a need to treat the impact pathways in greater
detail and substantiate them with empirical evidence.
Most previous concept papers and evaluations have been devoted to studying large-
scale research infrastructures, or so-called ‘big science’ facilities that are of European
and international significance. Empirical work on investment in mid-size
instrumentation of national and macro-regional significance barely exists. This is
mainly the result of the fact that RI investments have not yet been subject to regular
evaluations. There is a scope for further methodological developments and their
applications in evaluations to evolve from narratives of socio-economic effect of RIs to
more empirical models and measurement techniques. This knowledge would help to
move from simple ex-post detection of intended and unintended returns of RIs to a
better understanding and planning of future investment.

1.3 The aim of this guidance document

The aim of this guidance document is to disentangle and characterise the concrete
benefits of RI investment for different stakeholders and to build a schematic impact
assessment framework that can be used in evaluations to trace the core impact
pathways. Section two introduces a logical framework for the impact assessment of RI.
Recognising that each RI is embedded in specific socio-economic conditions, there will
never be a ‘one-size fits all’ approach to mapping all socio-economic impacts. This
work should be regarded as a step towards defining a typology of possible effects and
accounting for the conditions and patterns that enable their creation and diffusion.
The third section elaborates on the methods and tools that are suitable for the
measurement of respective impacts. The identified impact pathways involve complex
and multidimensional phenomena, which in most cases cannot be captured with
standard quantitative indicators, but require a thorough triangulation of quantitative
and qualitative evidence. The section elaborates on the appropriate indicators for each
type of impact and provides additional guidance on the respective data requirements,
their potential sources and advice on collection routines. Reflections on the limitation
and challenges of the application of this IA framework are outlined in conclusion.

4 Simmonds, P. et.al (2013) Big Science and Innovation. Report for the UK Department for Business,
Innovation & Skills
5 Multipliers - Further economic activity (e.g. jobs, expenditure or income) associated with the inputs to or
outputs from the project. According the European Commission (DG REGIO) an income multiplier is a
secondary effect resulting from increased income and consumption generated by the public intervention
(investment). Multiplier effects are cumulative and take into account the fact that part of the income
generated is spent again and generates other income, and so on in several successive cycles. In each cycle,
the multiplier effect diminishes due to purchases outside the territory. The effect decreases much faster
when the territory is small and when its economy is open.
6 Horlings, E. et.al (2012) The societal footprint of big science. Report of the Rathenau Instituut, the Hague,
the Netherlands
7 http://www.fenriam.eu/

Evaluating the socio-economic impact of research infrastructures

© Technopolis Group, 2015 3
2. Logical framework for socio-economic impacts of investment in
research infrastructure
Figure 3 provides a schematic overview of the various direct and indirect benefits that
arise during two main periods of a research facility’s lifecycle:
1) design and construction phase; and
2) operational phase of research facilities.
The specificities of each of the pathways and their contributions to other impacts that
are singled-out in this two-dimensional graph are described in detail in the following
sub-sections of this chapter. The types of impacts are positioned in the graph to
visually reflect better their relation to each other, not an order of their importance.
Societal impacts can be of very diverse nature and lead to broad, overarching
outcomes. In order to provide a concise graphical representation of IA framework
these impacts are not schematically reflected in this graph.
This IA framework is predominantly designed for single-sited and geographically
confined national or macro-regional distributed research infrastructures. With regards
to highly distributed and virtual RI case-specific a range of additional factors have to
be considered. These types of RI usually are decentralised administratively and
financially and include much larger geographical areas of impact. They can have
mainly or only virtual access without the requirement to cover physical support to
individual users. Advanced ICT-based technical solutions and well-designed access
rules may ensure much larger outreach with dispersed impact on user and stakeholder
groups. These characteristics of highly distributed and virtual RI require more case-
specific approach to impact assessment.

2.1 Design and construction phase

2.1.1 Economic impacts

During the design and construction phase a wide range of technical and scientific
knowledge is applied to set-up the required facilities. The physical construction and
refurbishment of buildings and their accompanying networks and utilities engages
many companies ranging from construction material suppliers to technical and legal
service providers. The procurement of core construction services typically involves
local companies8, which creates additional revenue and increases temporary
employment opportunities in the region. Despite the fact that the impact of many of
these expenditures is usually confined and short-lived for smaller-scale RI, there can
be a significant further multiplier effect on the local economy, for example, due
improved reputation credentials for firms that supply high-tech facilities and an
increase in the local tax base. In-depth analysis of the local industrial base and
mapping of relevant global supply chains is required to scope the total economic effect
of the RI investment.

8 Locally-owned businesses, franchise or corporate branch operating within a local area

Evaluating the socio-economic impact of research infrastructures

© Technopolis Group, 2015 4
Figure 3: Logical framework for socio-economic impact assessment of investment in research infrastructure

Evaluating the socio-economic impact of research infrastructure

© Technopolis Group, 2015 5
Figure 4: Indicators for economic impacts of RI construction phase
No Indicators Explanatory remarks and examples
1. Number of commercial suppliers Suppliers can be mapped per sector, field
for RI design and construction of activity, size, technology group (low-
phase tech/high-tech), geographic scope
(national/international/multinational)
2. Scale of commercial suppliers’ For applied method to estimate utilities in
turnover increase due to RI monetary terms see the study “Economic
utility resulting from CERN contracts”9
3. Scale of commercial suppliers’ Measured in FTE and grouped per
employment increase due to RI category of employees (e.g. technical staff,
R&D staff)

2.1.2 Impacts on innovation

Research facilities under construction may require specific design and building
solutions to serve the purposes of a particular research field (e.g. laboratories, clean
rooms, etc.). Such requirements foster collaboration among the involved suppliers,
facility mangers and scientists in developing innovative design solutions and building
functionalities. Local companies can benefit from accumulated know-how and
complementary skills that expand their competitive advantage in other markets (e.g.
enable winning international procurements of similar research facilities, secure service
contracts as external consultants to other developers). In cases where foreign
companies demonstrate a superior skills base and are selected to address particular
design and construction tasks, the benefits of the investment flow out of the national
economy. Moreover, in the RI construction phase, the universities and responsible
government agencies may acquire additional and more sophisticated project
management skills that can have knock-on effects on the potential for public sector
innovation.
The purchase of the necessary scientific instruments and supporting equipment is
generally open to international bidders, as the supplier base for such specific
commodities and accompanying services is often very specialised and, in certain cases,
the specific equipment may not exist ‘off-the-shelf’ on the market. Hence, the
procurement of equipment can have a considerable impact on extending the
technological frontier and fostering innovation10. As a result, the impact from this
procurement will be distributed across specific global supply chains for scientific
instrumentation.

9 Bianchi-Streit, M. et al (1984) Economic utility resulting from CERN contracts (second study). Report of
the European Organisation for Nuclear Research (CERN)
http://indico.cern.ch/event/66952/contribution/0/2/material/paper/0.pdf

10 For the theoretical background of this nexus see work of von Hippel, E. (1976) The Dominant Role of
Users in the Scientific Instrument Innovation Process, Research Policy 5 (3)

Evaluating the socio-economic impact of research infrastructure

© Technopolis Group, 2015 6
Figure 5: Indicators of innovation impact of RI construction phase
No Indicators Explanatory remarks and examples
1. Number of joint The novel technical requirements for the
development activities construction (design or building techniques) of
with suppliers scientific facilities or the equipment require
close interaction between the recipient
organisation and one or more suppliers. This
can be realised through a process of pre-
commercial public procurement11, design
contest, a forward commitment procurement or
a competitive dialogue. In these cases, the
recipient organisation defines specific technical
requirements or criteria to be met but not the
precise solution (in the form of a specific piece
of equipment, etc.).
2. Number of contracts The focus should be on the extent to which
concluded for high-tech or specific design or technical requirements for
specialist services that scientific buildings or equipment induce an
require development, or impact on learning and skill development
calibration of amongst supplier firms, notably local (national)
designs/equipment to firms.
meet specific requirements

2.2 Operational phase

Two main categories of impacts of the operational phase of RI exist: 1) impacts from
the routine operation, maintenance and upgrading of RI; and 2) impacts from the use
of research facilities.

2.2.1 Economic impacts

Benefits from RI operation include the assessment of the performance characteristics
of the research facilities. This includes a review of RI performance addressing aspects
of quality, access, reliability of the infrastructure, account of all the services provided
and assessment of the optimal usage of equipment. Among the criteria to review is the
flexibility of staff to support external users, feasibility of contractual arrangements for
granting access, quality of accompanying services (e.g. tech transfer, incubation
services). Performance aspects have a direct impact on the efficiency and effectiveness
of RI operations, user satisfaction, and consequently on the generated revenues to RI
and its users.
Maintenance and operation of RI involves longer-term effects on employment in
higher education establishments, science parks and companies, such as additional jobs
for scientists, technicians, administrative and support personnel working on the RI.
The operation of facilities also includes expenditure on goods and services and routine
upgrades with additional procurement design and associated equipment. These
activities have a further multiplier effect on the local economy and relevant global
supply chains.

11 http://www.innovatiefaanbesteden.be/files/409

Evaluating the socio-economic impact of research infrastructure

© Technopolis Group, 2015 7
Figure 6: Indicators for economic impact of RI operational phase
No Indicators
1. Number of scientists,
students, state-owned or
private enterprises that
benefitted from RI services
2. Total amount of funding
generated from services,
grants and joint projects
3. Number of new directly
and indirectly created jobs
4. Total amount of
expenditure on personnel,
operations, maintenance
5. Total RI capacity
utilisation
6. RI capacity utilisation
external business users
7. Financial sustainability of
RI
Explanatory remarks and examples
Service can be categorised into broad types,
such as: free or for a fee access to equipment
(including technical support from qualified
technicians), testing or metrology (carried out
by the RI staff in laboratories or using mobile
equipment), contract research, consultancy,
hosting doctoral students, etc.
Per service type, source of research grant and
type of joint project
Per category (scientific/ technical/
administrative) and wage level; the aim is to
scope how RI impacts on an increased
employability of human resources.
For centralised large-scale facilities the rule of
thumb is to calculate around 10% of investment
costs for annual operation expenditure. Yet for
most other types of RI this percentage is
significantly higher.12 For distributed RI there
are higher costs due to coordination,
management and consultation tasks requiring
qualified personnel. Operating costs for RI in
the humanities and social sciences are usually
the most significant expenditure.13
RI access hours used as % of the total available
access time
RI access hours used by business entities as % of
the total available access time
Measured as % of the total costs funded from
the provided services, received grants and
realised joint projects

2.2.2 Impacts on human resource capacity

The build up of human resource capital is a major benefit of RI. Modern research
facilities can serve as magnets to retain and attract talent. This includes local
researchers, technicians, students and global talent through recruitment of permanent
staff and the promotion of the flow of visiting researchers and foreign students.
RI has a crucial role to play in training and skills development. Research facilities
concentrate skilled staff that hold tacit knowledge how to operate them. Only with the
help of skilled technicians and experienced researchers that students and other
interested stakeholders can learn to set up experiments and interpret their results.
This tacit dimension of knowledge transfer makes RIs entry points into networks of
knowledge, expertise and practice. To determine the impact of RI on skills
development it is important to explore how the exchange between the younger staff

12 Wissenschaftsrat (2013) Report on the Science-Driven Evaluation of Large Research Infrastructure

Projects for the National Roadmap (Pilot Phase)
13 Wissenschaftsrat (2011) Recommendations on Research Infrastructures in Humanities and Social

Sciences

Evaluating the socio-economic impact of research infrastructure

© Technopolis Group, 2015 8
(MAs, PhDs, post-docs) and experienced researchers and technicians is structured and
promoted. Equally important is to appraise the ways tacit knowledge is transferred to
private sector users.
The diffusion of knowledge gained by the RI management, operators and users may
have a significant societal value. For example, the research commercialisation skills
acquired by the management and scientists at a RI is an important benefit with far-
reaching implications on the local innovation system. Closer formal and informal
social networks increase interpersonal trust and knowledge sharing. This consequently
augments the quantity and diversity of knowledge that is available to RI users. Social
networks create new forms of interaction among the actors of the innovation system,
stimulate learning environments and increase the awareness of users (potential and
existing) about the scope of knowledge that is available on RI. In short, social capital
enables the full exploitation of accumulated human capital.
The mobility of people and emergence of new cooperation networks is a major
mechanism through which knowledge circulates and diffuses the benefits to wider
economy. Graduates leave RI with the knowledge of the most recent scientific results,
skills using advanced instrumentation and ability to apply them to complex problem
solving. Internships and recruitment of graduates mean that students act as a form of
‘social glue’ and they often create long-term links between scientists and businesses.14
However, if the local academic and industrial environment does not provide sufficient
absorption capacity for these advanced skills, the accumulated human capital may
remain underutilised or emigrate.
Figure 7: Indicators for impact of RI operation on human resource capacity
No Indicators Explanatory remarks and examples
1. Number of new jobs for Per category (permanent/visiting) and per year;
research and technical staff the aim is to understand the influx of new
attracted from abroad as % knowledge that is directly linked with RI
of the total number of staff
employed on RI
2. Number of Master thesis Per year of completion and per scientific field;
defended, where Master theses are considered as a
knowledge and skills training/education impact on skills profile of
gained on RI were population; the number of PhD thesis
exploited completed is counted in the impacts on
scientific activity
3. Number of graduates Per stage (MA, PhD) and per year; distinction
trained on RI should be made on skills gained by PhD
students that improve their general academic
and employability profile and skills directly
applied in PhD thesis work; the latter should
account for impacts on scientific activity
4. Number of foreign Per stage (MA, PhD) and per year
students as % of all
students trained on RI
5. Data on the post-diploma Descriptive data should include at least
employment path of those indication on the share of graduates finding
graduates trained on RI employment in academia/public sector/private
sector; further statistics on types of graduate
employment (part time/full time) and sector of
activity are desirable

14 Bozeman, B. (2000) Technology transfer and public policy: a review of research and theory. Research

Policy, 29(4–5)

Evaluating the socio-economic impact of research infrastructure

© Technopolis Group, 2015 9
2.2.3 Impacts on innovation
Innovation can result from RI through industry collaborations with academic research
groups and contract research that result in knowledge transfer. Research facilities
generate knowledge that businesses cannot acquire through their networks. More
exposure to RI knowledge networks through formal and informal social interaction
increases the possibility that private sector discerns new opportunities for exploiting
intellectual and technical capital available on RI.
Spin-offs are one way in which knowledge generated using RI can diffuse into the
market in the form of innovative product, process or service. Other avenues for
innovation impacts arise from the commercialisation of research through licences or
joint ventures with existing companies.
There are important barriers for private sector use of RI due to different objectives and
‘cultural factors’. The industrial community is often less interested in using RI directly
(e.g. placing industrial researchers in RIs to carry out R&D). There is more interest in
acquiring technical training and co-developing new IPR.15 In some sectors (like food or
agriculture) the industrial landscape can be dominated by a large number of SMEs
that, individually, lack resources to engage in research collaboration.
The table below provides a number of indicators that can be used to track the impact
of RIs on knowledge transfer and innovation. One example of a framework for
measuring the quantity of knowledge transfer from the research base is the Scottish
Funding Council's (SFC) Knowledge Transfer Metrics Return16.
Figure 8: Indicators for impacts on innovation of RI operational phase
No Indicators Explanatory remarks and examples
1. Number of collaborative Collaborative research carried out at that the RI
research projects and might involve both regional and international
volume of funding researchers using the equipment for shorter-or-
longer periods. The funding generated from
service fees charged to research team will enable
reinvestment in equipment. Qualitative research
on the research teams working in or with the RI
can be used to map (e.g. using social network
analysis) the place of the RI in global research
networks.
2. Number of R&D projects Contract research is one step closer to
commissioned by generating a socio-economic impact since firms
companies and volume of purchasing research services will only do so in
their funding order to improve production processes
(enhancing productivity) or develop prototypes
of new products or services. Tracking trends in
volume of funding provides an indicator of the
longer-term financial sustainability of the RIs.
Similarly, tracking the renewal of the ‘client
base’ (contracts with new clients) rather than
just number of companies contracting research
is useful.

15 ERF Workshop “The Socio-Economic Relevance of Research Infrastructures”, 31 May – 1 June 2012,
Hamburg, Germany
16 The metric is used by the Scottish Government to measure as a proxy measure of the quantity, but not the
quality, of knowledge exchange activities undertaken by Scottish universities.
http://www.gov.scot/About/Performance/scotPerforms/TechNotes/knowledge

Evaluating the socio-economic impact of research infrastructure

© Technopolis Group, 2015 10
 No Indicators
3. Number of technologies,
prototypes, industrial
designs developed and
transferred
4. Number of start-ups and
spin-offs created with
support from RI services –
growth in turnover/value
added and employment
5. Number of feasibility or
market studies for
industrial investment and
application of technologies
6. Procurement contracts
signed for development
and upgrade of research
equipment
Explanatory remarks and examples
Depending on the RI’s intellectual policy policy,
the results of collaborative and contract
research may be potentially exploited through
proof of concept and prototyping either by the
RI itself or in co-operation with academic or
industrial partners. Ideally, the monitoring of
outcomes should go beyond ‘number counting’
and aim to track whether prototypes have been
scaled up or further developed commercially.
The number of start-ups/spin-offs is an
indicator commonly used to assess economic
impact. However, there is a need to avoid
unrealistic expectations when setting targets for
numbers of spin-offs as this involves IP transfer
to the new companies potentially closing future
avenues of academic research. Moreover, many
spin-offs fail to grow substantially and there is
an need to track growth in turnover and
employment on a longitudinal basis (over 5-10
years) to fully understand impact.
Commercialisation of research results will
involve, traditionally, market research or
industrial scale feasibility study prior to
investment. The number of (e.g. proof of
concept/prototyping) projects that move to the
stage of industrial investment is a strong
indication of the economic relevance of the
research. Tracking over time (5-10 years) the
actual investment in the application of new
technologies by businesses provides a stronger
indication of economic impact.
Tracking the number and type of procurement
contract for the development of innovative
instruments and products (rather than simply
‘off-the-shelf purchases from equipment
suppliers provides an indication of the impact of
the RI on the scientific equipment market – this
is particularly pertinent if suppliers are
regionally based.

2.2.4 Impacts on scientific activity

The impact of RI on scientific activities can result in accumulation of new knowledge
and methodologies to push the boundaries of fundamental science. There is an
inherent tension between the development of non-proprietary research, which is
published in relevant scientific journals and can be accessed by other interested
parties, and propriety research where knowledge and technical data, which can be
basis for specific inventions, are not disclosed openly.
Another linkage to scrutinise is the impact of the use of RI on scientific productivity,
international recognition and reputation. The availability of advanced equipment has
an effect on shaping how scientific communities organise themselves. It can
significantly increase the productivity of research teams, as they do not need to seek
and arrange experimentation and testing opportunities on limited-availability facilities
or abroad (e.g. access to Grid computing facilities substitutes the need to queue for
supercomputer service time).

Evaluating the socio-economic impact of research infrastructure

© Technopolis Group, 2015 11
Open access to RI also induces more regular inward flow of researchers promoting
closer involvement of local teams in international research networks (brain exchange).
Such interaction can generate important learning effects for experienced and early-
stage researchers alike. Increase in international recognition of leading scientists and
the capacity of research teams can attract further international competitive funding to
the research system. In addition, researcher experience gained during the set-up and
operational phase of RI can have further policy impact through contributions to RI
roadmaps.
Figure 9: Indicators for impact on scientific activity
No Indicators Explanatory remarks and examples
1. Number of articles The aim should be to track the dynamics of
published in the ISI level publishing behaviour due to RI; peer-reviewed
international scientific articles applies as an indicator of scientific
journals as a direct result activity to most scientific fields; in humanities
of research using RI and some sub-fields of social sciences books,
book chapters and monographs are more
relevant indicator
2. Number of Applies to those scientific fields where this
methodologies/designs output is relevant to measure scientific
developed productivity
3. International patents Indicator is relevant for most scientific fields in
granted and published natural sciences, engineering, life sciences and
patent applications (all medicine; in humanities and social sciences
types) other types of IPR (e.g. copyrights) should be
considered
4. Number of PhD Per year of completion and per scientific field;
dissertations completed includes those PhD dissertations which have
been predominantly or partly-based on the use
of RI services or equipment
5. Number of scientific events Data on the frequency, types of events,
organised on research geographic origin and institutional affiliation of
topics directly relating to participants can provide some proxy on the RI
RI services impact on mutual learning, knowledge exchange
and researcher involvement in major scientific
networks; some conclusions can be drawn on
the critical mass and synergy effects with other
RI facilities in the region

2.2.5 Broader impacts on society

Regarding wider social impacts, it is important to outline that research infrastructures
can play an important role in scientific communication and scientific education. RI
can be used to inspire school students to learn STEM subjects. For example, large-
scale RI may organise open days for the general public or for schools. The policy-
decision on investment in RI can also be widely reflected in the press leading to
increased public awareness of science.
Research outcomes from using RI services can lead to concrete innovative products,
and services that are taken up and diffused in society (for example, new medical
instruments, diagnostics, treatments). While such impacts on society are broad,
indirect and very difficult to attribute and quantify, it is nevertheless important to
scope the potential contribution of RI to solving various societal problems.
There can also be concrete benefits that result from improving local infrastructure,
urban planning and community services as the RI investment may revitalise certain
areas with important indirect societal benefits. These developments can further shape
local cultural activities and citizen lifestyles.

Evaluating the socio-economic impact of research infrastructure

© Technopolis Group, 2015 12
Figure 10: Indicators for impacts on society
No Indicators
1. Number of organised RI open
days for wider public and any
available data on participant
satisfaction with the events
2. Number of press articles on the
investment in research
infrastructure
3. Number of new or improved
products, services, solutions as a
result of research using RI that
are diffused in society
4. Account of improved local
infrastructure, community
services, increase in local
cultural/recreational activities
due to RI
Explanatory remarks and examples
Per year and per target group (e.g.
schools/general public); aim to provide an
indication on the impact of public
awareness of science
Per year and per type of publication (print
media/online articles/specialist
publications)
This indicator involves especially
significant time-lag; the aim to scope the
possible contribution of RI to specific
societal outcomes
Aim to scope the possible contribution of
RI to the outcomes

3. Impact assessment methods and tools

3.1 Impacts on economy
To measure the direct economic effect of the RI design and construction phase
requires an analysis of all suppliers that have won the procurement tenders to scope
the contract values per sector of activity and service provider affiliations
(local/foreign/multinational). Relevant supply chains should be mapped to determine
the scope of the output, value added and additional employment that has been created
locally and the benefits that have been diffused internationally. Desk analysis of the
proposal and contractual documentation and survey among suppliers can be used to
collect this information.
The operational phase requires a comprehensive look at the activities undertaken by
the managers responsible for establishing and running the research facility. This
involves a detailed account of facilities expenditure on salaries, purchase of goods and
services from financial records and direct output generated. Desk research and
interviews with RI managers/coordinators can be used to collate and analyse financial
and activity data.
The multiplier effect on the economy is addressed through an input-output analysis.
This ‘ripple effect’ includes the indirect economic effects that occur from all secondary
industries supporting the procured activities and induced effect representing the
changes in consumption of respective households. The input-output model is a
quantitative econometric technique that represents the interdependencies between
different sectors of a national/ regional economy. An input-output analysis estimates
the value of the purchases that flow between the identified supply chains in a given
period of time. Building IO model it is possible to derive economic multipliers.
Methodological approaches have been developed on applying cost-benefit analysis to
RI projects17, yet these are suited more for ex-ante project evaluation as it takes a long
time period for the effects of RI investment costs to materialise. Nonetheless, the
applicability of CBA type approaches can be explored. For instance, the effects on RI

17 See e.g. Clarke, S. et.al. (2013) Project Preparation and CBA of RDI Infrastructure Projects. JASPERS
Staff Working Paper

Evaluating the socio-economic impact of research infrastructure

© Technopolis Group, 2015 13
users can be scoped through application of contingent valuation techniques.
Contingent valuation is a method that uses surveys to draw out the willingness of
actual and potential users to pay for certain services. The method involves assigning of
monetary values to non-market goods and services based on preferences.
Data requirements
• Overview of contractual information with RI suppliers (design, construction,
maintenance, operations, renewal)
• Data on the affiliation of suppliers (local/foreign/multinational)
• Data on employment, wages
• Facilities annual expenditure broken down by function in time series
• Strategic plans and annual activity reports

3.2 Impacts on human resource capacity

To account for the impacts on human resource capacity it is necessary to capture the
structural changes in organisational behaviour that arise as a result of the research
infrastructure. Monitoring statistics on the number of staff and students attracted, in
particular from abroad, can shed a light on the dynamics of knowledge exchange that
RI is facilitating. It is necessary to understand what are the formal organisational
aspects, how training on RI is carried out and what are the practices for informal
know-how transfer (interaction with technicians, networks). Surveys among students,
vising and permanent researchers, technicians and RI managers can be employed for
necessary data gathering. Particularly important is to explore the spaces of student
and researcher interaction with industry. Longitudinal analysis of graduate career
paths is a valuable approach.
In addition to the available monitoring statistics and information from surveys, data
collection strategies for tracing impacts on human resource capacity should involve
extensive qualitative research techniques. These can include semi-structured
interviews with facility owners and a representative population of researchers and
students. A case study approach is an appropriate way to capture the dominant
patterns of knowledge diffusion and the degree of social capital formation that the
establishment of RI has promoted.
Data requirements
• Relevant monitoring data, including e.g. statistics on the numbers of attracted staff and
students, (locally and from abroad) and career paths of graduates that have used RI for
training or MA/PhD thesis
• Database of concluded service contracts
• Documentation of human resource development strategies, user guides and other
documented evidence
• Annual activity reports

3.3 Impacts on innovation

Impacts on innovations arise via multiple impact pathways. The most feasible method
is to apply quantitative analysis to scope economic effects of innovation (e.g. value of
IP portfolio) and use qualitative investigation to understand in detail the knowledge
spillovers. Monitoring data on knowledge transfer activities can be supplemented with
supplier surveys (for tracing procurement-led innovation and skills formation) and in-
depth interviews (with research teams, spin-offs, relevant industry representatives).
Narrative descriptions of individual case studies that look in detail into specific aspects
of innovation impact (e.g. impact via prototyping, industrial demonstrators) would be
supportive for the overall assessment. If any routine user surveys are executed by RI
management this would be very useful source of material to better understand user
experiences, motivations and wider effects of RI use.

Evaluating the socio-economic impact of research infrastructure

© Technopolis Group, 2015 14
 Data requirements
• Relevant official statistics and structured monitoring data (i.e. regular progress reports
and data for measure project level key performance indicators)
• Database of users (academic, industrial) over the lifetime of the facility
• Database of supplier contracts (held either by the funding agency or by beneficiary
higher education or public research institutes)

3.4 Impacts on scientific activity

Impacts on scientific activity can be analysed through a bibliometric analysis of the
research teams working on the RI. Especially valuable would be to see the evolution of
co-publications over a period of time. A time-series/network analysis of the
participation in international research project consortia (FP7, H2020) may shed some
light on the visibility of research groups on the international research landscape.
All quantitative techniques should be supplemented with in-depth interviews with key
researchers to deepen, test and validate the claims. There are many limitations, as the
knowledge spillovers that arise as a result of the intellectual advances may be largely
invisible to facility owners and stakeholders.

Data requirements
• Database of researchers working on the RI
• Relevant official statistics and structured monitoring data
• Annual activity reports

3.5 Impacts on society

The broader impacts that research infrastructure investment have on society is the
least scrutinised area due to the fact that such impacts are difficult to trace and
quantify. While direct causal chains are almost impossible to establish, social science
methods can be employed to estimate the potential contribution of RI. Surveys among
schools, public sector, relevant stakeholder groups can be used for this purpose.
Further interviews with relevant target groups for RI communication and PR activities
and the involved researchers can be undertaken. Impact on STEM education and
public understanding of science can be best explained through analytical narrative in
the form of case studies.
Data requirements
• Documented materials on RI public relations activities (open days, public lectures,
seminars)
• Available data from satisfaction and feedback surveys of participants to RI
communication events
• Track of media publications/online content regarding RI

Evaluating the socio-economic impact of research infrastructure

© Technopolis Group, 2015 15
Figure 11: Implementation of an impact assessment of research infrastructure

Evaluating the socio-economic impact of research infrastructure

© Technopolis Group, 2015 16
3.6 IA implementation process, its challenges and limitations
Figure 11 provides an overview the IA implementation steps and use of analytical
techniques. During the inception phase desk research is performed on all available
documented evidence to determine the information gaps that should be covered
through additional data collection methods. In the data collection phase surveys are
launched beforehand. In-depth interviews and the application of quantitative
techniques can be run in parallel. Case studies are usually chosen after the first
preliminary findings to ensure that the most relevant aspects and impact areas are
scrutinised in detail. The synthesis phase of findings ideally should involve a peer
review and an international expert team to ensure appropriate benchmarking of RI
performance. A full impact assessment takes usually around six to nine month period.
Data shortages are likely to be the most important challenge due to the limited data
caption routines during the lifetime of research facility. Database development with
appropriate quantitative data is a foundation for carrying out an in-depth impact
assessment. This calls for anticipatory collection of relevant data at an early stage of RI
project implementation. The principle should be that the amount if work for
researchers is kept to a minimum. RI management should be encouraged to regularly
assess key sources of information and collect relevant data on an on-going basis. These
efforts should include as a minimum data which is readily available or which can be
collected automatically, e.g. CVs, completed doctorates, concluded contracts, etc.
Bearing in mind the challenge of data availablity and quality, the expectations have to
be realistic in terms of the quantifiable findings that can result from the evaluation.
Further challenges and limiting factors to IA of RI include:
• The difficulty of capturing multidimensional and complex research outcomes and
impacts with simple metrics that are used as core indicators. For instance, the
accuracy of input-output models to determine multipliers is questioned. Findings
from the application of quantitative methods should be well-balanced with
qualitative insights and domain expert validation.
• Issue of displacement – or the extent to which economic benefits occur at the
expense of other economic activities – is hard to address in the analysis.
• Delineation between short-term, medium and long-term effects will be very
difficult due to restraints in data availability in time series.
• It is hard to separate between RI induced benefits and those fostered by other
determinants. The impacts of RI should be accompanied by analysis of parallel
initiatives in higher education and support measures for enterprises. National
legal frameworks (e.g. rules for skilled migration, IPR) are all important
determinants.
The establishment of systematic evaluation practice for investment in research
infrastructures is key to expand reference material and foster learning for more
comprehensive analyses of RI impacts.

Evaluating the socio-economic impact of research infrastructure

© Technopolis Group, 2015 17
Bibliography

Autio, E. (2014) Innovation from Big Science: Enhancing Big Science Impact Agenda.
Report to UK Department for Business, Innovation & Skills. Available at:
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/28
8481/bis-14-618-innovation-from-big-science-enhancing-big-science-impact-
agenda.pdf
Autio, E., Streit-Bianchi, M., and Hameri, A.-P. (2003) Technology Transfer and
Technological Learning Through CERN’s Procurement Activity, CERN Scientific
Information Service. Geneva
Bianchi-Streit, M. et al (1984) Economic utility resulting from CERN contracts (second
study). Report of the European Organisation for Nuclear Research (CERN). Available
at: http://indico.cern.ch/event/66952/contribution/0/2/material/paper/0.pdf
Bozeman, B. (2000) Technology transfer and public policy: a review of research and
theory. Research Policy, 29(4–5)
Clarke, S. et.al. (2013) Project Preparation and CBA of RDI Infrastructure Projects.
JASPERS Staff Working Paper. Available at:
http://www.jaspersnetwork.org/display/for/Project+Preparation+and+CBA+of+RDI
+Infrastructure+Projects
European Strategy Forum on Research Infrastructures (2011) Evaluation Report 2011.
Available at:
http://ec.europa.eu/research/infrastructures/pdf/esfri_evaluation_report_2011.pdf
Fotakis, C. (2010) Analyses of FP7 supported Research Infrastructures initatives in the
context of ERA. Available at:
http://ec.europa.eu/research/evaluations/pdf/archive/fp7-evidence-
base/experts_analysis/c.%20fotakis_-_research_infrastructure.pdf
Hallonsten, O., Benner, M. and Holmberg. G (2004) Impacts of Large-Scale Research
Facilities- A Socio-Economic Analysis. A study done at the Research Policy Institute,
Lund University. Available at:
http://rifi.gateway.bg/upload/docs/public_doc_REPORT_impact_of_large_scale_R
I.pdf
von Hippel, E. (1976) The Dominant Role of Users in the Scientific Instrument
Innovation Process, Research Policy 5 (3)
Hickling Arthurs Low Corporation (2013) Return on Investment in Large Scale
Research Infrastructure. Report for National Research Council Canada. Available at:
http://www.triumf.ca/sites/default/files/HAL-ReturnOnInvestmentStudy-May-
2013.pdf
Horlings, E. et.al (2012) The societal footprint of big science. Report of the Rathenau
Instituut. Available at: http://www.rathenau.nl/en/publications/publication/the-
societal-footprint-of-big-science.html
National Research Council Canada (2013) Return on Investment in Large Research
Infrastructure. Available at: http://www.triumf.ca/sites/default/files/HAL-
ReturnOnInvestmentStudy-May-2013.pdf
OECD (2014) International Distributed Research Infrastructures: Issues and Options.
Available at: http://www.oecd.org/sti/sci-tech/international-distributed-research-
infrastructures.pdf
OECD (2014) Report on the Impacts of Large Research Infrastructure on Economic
Innovation and on Society: Case Studies at CERN. Available at:
http://www.oecd.org/sti/sci-tech/CERN-case-studies.pdf

Evaluating the socio-economic impact of research infrastructure

© Technopolis Group, 2015 18
 Rizutto, C. and Wood, J. (eds.) (2013) RAMIRI Handbook. Deliverable of FP7 project
Realising and Managing International Research Infrastructures. Available at:
http://www.ramiri-blog.eu/index.php?n=Main.HomePage
Rizzuto, C. (2012) Benefits of Research Infrastructures beyond Science, presentation
at ERF Workshop “The Socio-Economic Relevance of Research Infrastructures”, 31
May-1 June 2012, Hambourg
Sallee, C.M., Watkins, S.D and Rosaen, A.L. (2011) The Economic Impact of Fermi
National Accelerator Laboratory. Anderson Economic Group report to the University
of Chicago. Available at:
https://ovprnl.uchicago.edu/sites/research.uchicago.edu/files/Fermilab_Economic_I
mpact_Full_Study.pdf
Simmonds, P. et.al (2013) Big Science and Innovation. Technopolis Group report for
the UK Department for Business, Innovation & Skills. Available at:
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/24
9715/bis-13-861-big-science-and-innovation.pdf
Wissenschaftsrat (2011) Recommendations on Research Infrastructures in
Humanities and Social Sciences. Available at:
http://www.wissenschaftsrat.de/download/archiv/10465-11_engl.pdf
Wissenschaftsrat (2013) Report on the Science-Driven Evaluation of Large Research
Infrastructure Projects for the National Roadmap (Pilot Phase). Available at:
http://www.wissenschaftsrat.de/download/archiv/2841-13_engl.pdf
Zuijdam, F. et. al (2011) The role and added value of large-scale research facilities.
Technopolis Group report. Available at: http://www.technopolis-group.com/wp-
content/uploads/2011/02/1379_Report_Large-scale_Research_Facilities_EN1.pdf

Please cite this publication as:

Griniece E., A Reid and J. Angelis (2015): Guide to Evaluating and Monitoring Socio-
Economic Impact of Investment in Research Infrastructures. Technopolis Group.
Tallinn. Estonia

Front cover photo: Centre for New Pharmaceutical and Health Technologies of the
Santaka Valley, Kaunas, Lithuania. Source: http://www.mitnija.lt/

Evaluating the socio-economic impact of research infrastructure

© Technopolis Group, 2015 19

View publication stats