Science and Public Policy, volume 28, number 4, August 2001, pages 259 –266, Beech Tree Publishing,
10 Watford Close, Guildford, Surrey GU1 2EP, England
Productivity indicators
Sense and nonsense of S&T productivity
indicators
Rémi Barré
This article presents a process for benchmarking
scientific productivity fit for public policy pur-
poses, in which quantitative indicators have the
major role of initiating and focusing the discus-
sions. It shows that the calculation of a research
productivity indicator defined as the ratio of the
output (publications) of basic research to its
spending does not provide in itself any sensible
comparison among countries. It suggests that
quantitative indicators are useful as entry points
into discussions, considering that their raison
d’être is to be criticised in terms of their (limited)
relevance and (limited) comparability. In this
interpretation, benchmarking necessarily in-
volves direct interaction among people in an org-
anised process.
Rémi Barré is at Director of the Observatoire des Sciences et
des Techniques (OST), 93 rue de Vaugirard, 75006 Paris,
France; Tel: +33 1 42 22 30 30; Fax: +33 1 45 48 63 94; E-mail:
barre@obs-ost.fr. He is also Associate Professor at CNAM.
The second and third parts of this paper are partly based on
the results of an OECD project on Benchmarking Industry–
Science Relations, involving the author and Jean Guinet
(OECD).
Science and Public Policy August 2001 0302-3427/01/040259-8 US$08.00  Beech Tree Publishing 2001
ENCHMARKING PRODUCTIVITY be-
B tween different research systems may be an
important tool for comparing and improving
management practices and research policies. This
seems all the more interesting as the computation of
a productivity indicator for research seems very
simple: bib liometric measurements represent scie n-
tific output indicators, resources spent are input in-
dicators, their ratio would be the measurement of
scientific productivity.
The first part of this paper shows, through an ex-
ample, that interpreting directly such a ratio as the
productivity measurement of a research system is
meaningless and that this ratio alone cannot allow
for any conclusion that involves a comparison be-
tween research systems.
The second part of the paper suggests that what
the example demonstrates is very much in line with
what we have learned about research and innovation
processes in the last few years, namely that they in-
volve systemic and interactive dynamics among a
variety of actors. Thus any relevant concept of prod-
uctivity cannot be characterised by the ratio of one
output variable amongst many, to one input variable
amongst many. In fact, benchmarking scientific and
technological productivity needs to navigate through
a difficult terrain between two equally unacceptable
extremes: accepting that comparisons are impossible
or coming up with nonsensical oversimplifications.
The third part of the paper starts by reconsidering
the role of quantification and proposes to disregard
any mechanistic direct linkage between indicators
and productivity assessment. It introduces the idea
of an informed pluralistic debate in a wider network,
involving decision-makers. It suggests that quantita-
tive indicators of input and output and their ratios
259
Sense and nonsense of S&T productivity indicators
are useful, but only as entry points into the discus-
sion, considering that their raison d’être is to be
criticised in terms of their (limited) relevance and
(limited) comparability. I argue that such informed
criticism provides an excellent way to deepen the
understanding of the mechanisms at play in a com-
parative way.
The conclusion is what I consider to be a practical
and methodologically sound position regarding
benchmarking the productivity of research systems,
and a proposal for an exercise of benchmarking S&T
productivity which would make sense.
Nonsensical me asurement of productivity
In an article published by Science, Sir Robert May
(1998), at the time Chief Scientist of the British
Government, proposed an “indicator of efficiency of
public research spending”, constructed in the follow-
ing way:
number of scientific publicatio ns 1 similar in France and the UK (about 4.5 bn Euro)
spending on basic research
In 1996, the value of the indicator, as computed by
Sir Robert May, was 2.13 times higher for the
United Kingdom than for France.2 His conclusion
was that public research spending was 2.13 times
more efficient in the UK than in France. The case
included all the ingredients of an apparently perfect
case of benchmarking scientific productivity: an ex-
plicitly defined indicator of productivity; an inter-
national comparison; and a policy context at the
highest level. This section aims to illustrate that the
conclusion drawn is in essence illegitimate: the ratio
of ‘number of publications’ to ‘spending’ cannot be
interpreted as representing the efficiency of public
research spending.
What is ‘spending on basic research’?
Sir Robert defines as ‘basic research spending’ the
total spending of university research, government-
funded laboratories and non-profit research acti-
vities. It means that he measures in reality the
spending of all publicly funded research, which
includes academic, but also military and applied re-
search. Clearly, these latter kinds of research hardly
Table 1. Spending in basic research — initial and corrected values (1996)
Country Academic Other public research
research spending institutions spending
(a)
France 4.60
UK 4.25
Note: unit: bn Euro (109 Euro)
Source: for a and b: OST S&T Indicators Report 1998, based on OECD data
260
produce articles to be found in the Science Citation
Index (SCI), if at all.
We argue that the least that can be demanded for a
productivity indicator is that the input (resources
used), as denominator of the ratio, is effectively used
to produce the output placed as numerator! It is not
the case here, since part of the resources considered
as input (military and applied research) have nothing
to do with what is considered to be their output (SCI
publications). We must therefore restrict the meas-
urement of resources spent (the denominator), to the
spending for research which effectively produces
SCI-type publications.
Correction for factor of bias 1: Compute basic re-
search spending as academic 3 research spending,
plus 10% of all other public research spending, con-
sidering that only a fraction of the applied and mili-
tary public research can generate SCI-type public a-
tions (see Table 1).
It appears that spending on university research is
and that funding of other public research institutions
is quite different between the two countries, France
spending 50% more than the UK (6.25 against 4.19
bn Euro). Basic research spending as computed in
the Science article, was 28% higher in France than in
the UK. Following the correction, a more realistic
12% difference emerges.
Measuring scientific publications
Sir Robert considers the number of publications ap-
pearing in the SCI for each country as a comparable
measurement of their scientific production, but this
would hold true only if the world’s average re-
searcher in each discipline were to publish the same
number of articles, that is, if all disciplines had the
same tendency to publish in SCI journals. This is not
the case. Clinical medicine, for example, represents
30% of the number of articles in the SCI database,
for an estimated 10% of the spending in basic re-
search. This means that the tendency to publish in
that discipline is three times the average of the
others.
This would not result in a bias in the comparison
between France and the UK if the two countries had
the same level of activity in clinical medicine within
their disciplinary structure. This is not the case. The
Basic research spending as Corrected basic research
computed by May spending
(b) (c = a + b) (d = a + 0.1b)
6.25 10.85 5.23
4.19 8.44 4.67
Science and Public Policy August 2001
 Sense and nonsense of S&T productivity indicators

Table 2. World share and absolute number of articles for clinical medicine, other disciplines and all disciplines, for France and the
UK (1996)

Country World share of articles Absolute number of articles

(%) (world total = 100)
Clinical Other All Clinical Other All
medicine disciplines disciplines medicine disciplines disciplines

France 4.8 5.2 5.1 1.45 3.65 5.1

UK 12.0 7.0 8.5 3.6 4.9 8.5
World 100 100 100 30 70 100

Table 3. Number of articles for clinical medicine, other disciplines and all disciplines, for France and the UK with correcting factor
for clinical medicine (1996)

Country Absolute number of articles Absolute number of articles

(world total = 80) (world total = 100)

Clinical medicine Other disciplines All disciplines

France 0.48 3.65 4.13 5.2

UK 1.20 4.9 6.10 7.6
World 10 70 80 100

UK is strongly specialised in clinical medicine, with
a share of 12.0% of world articles, which is 41%
above its average (all disciplines). On the other
hand, the world share of articles in clinical medicine
for France is 5% below its average. The high ten-
dency of publication in clinical medicine and the
high specialisation of the UK in that discipline result
in a structural effect which constitutes a bias in the
comparison of the scientific outputs between France
and the UK.
How can this bias be controlled for? I separate
clinical medicine and the ‘other disciplines’ which,
together, make for ‘all disciplines’. This provides
three categories of discipline. Let us first consider
the world share of articles for both countries in each
of these categories (Table 2, columns 1 to 3). Then,
knowing that clin ical medicine makes for 30% of the
SCI database, I compute the absolute number of arti-
cles from each country for each category, applying a
coefficient of 30% to the category clinical medicine,
70% to other disciplines and 100% to the category
all disciplines (Table 2, columns 4 to 6). By con-
struction, the numbers for ‘all disciplines’ are the
same in columns 3 and 6.
At this stage, I apply a coefficient of one-third to
the absolute number of clinical medicine articles, to
account for the fact that the tendency of publication
in this discipline is three times higher than in the
others (Table 3, column 1). The numbers for ‘other
disciplines’ are kept (column 2), which gives a new
total for each country, with a world total of 80. Fi-
nally, I adjust the total of each country to get the
figures for a total of 100, which gives their world
share, after correction for the structural effect of
clinical medicine (column 4).
Correction for factor of bias 2: To eliminate the bias
introduced in the measurement of scientif ic output
by the high tendency of publication of clinical medi-
cine and of the high specialisation of the UK, I
introduce a weighting factor of one-third for the arti-
cles of that discipline (Table 4).
The scientific output of the UK was 67% higher than
that of France with the initial numbers; it is 46%
higher after introduction of the correction needed to
take into account the clinical medicine bias.
First round correction of productivity indicator
In Table 5, the initial and corrected values for the
productivity indicator are computed. The initial
value computed with OST (Observatoire des Sci-
ences et Techniques) data fits very well with the one
used by Sir Robert, since the indices are almost
equal (213 and 214). Using now the corrected values
for the denominator and the numerator, following
the first round corrections, we get a corrected value
for the productivity indicator. This is 164, which in
Table 4. Volume of scientific publications — initial and
corrected values (1996)
Country SCI publications SCI publications as% of
as% of world world total after correction
total for clinical medicine
(e) (f)
France 5.1 5.2
UK 8.5 7.6
Sources : OST S&T indicators Report 1998 ; from ISI–SCI data

Science and Public Policy August 2001 261

Sense and nonsense of S&T productivity indicators

Table 5. Productivity value after first round of corrections

Country Indicator directly derived Indicator computed by OST b Indicator computed with corrected
from those published keeping the definitions values eliminating factors of
by Maya of May bias 1 & 2
(g = e/c) (i = f/d)

France 100 100 100

UK 213 214 164
a
Notes: recomputed to give the reference value 100 for France
b
with OST data

Sir Robert’s terms would mean that UK public ac-
tivities in science and technology would be 64%
more productive than their French counterparts. Is it
certain that this residual 64% represents a difference
in productivity?
Second and third round of corrections
In the first round of corrections I took into account
two factors of bias. There are, however, a number of
other factors that introduce a difference between
France and the UK in the numerical value of the
productivity indicator without having anything to do
with the productivity of research at all. I now con-
sider another four such factors (numbers 3 to 6) in a
second round of corrections, to eliminate the biases
they introduce. I do this through making explicit
simplifying assumptions, modifying directly the
ratio, without detailed presentation of the new
values for the numerator and/or the denominator
(Table 6).
Factor of bias 3: unit cost for salary and social
benefits4 : The majority of researchers in UK univer-
sities work under short-term contract (‘post-docs’)
with rather limited social benefits. The situation is
quite different in France, where almost all academic
researchers are civil servants, with significant social
benefits.
Factor of bias 4: weight of social sciences and
humanities: The SCI does not include social sciences
and humanities. Therefore, we must discount from
the spending (the denominator), the resources de-
voted to this field. They represent a share of the
spending on basic research in France, which is
probably higher than in the UK.
Factor of bias 5: linguistic and cultural: The
hypothesis here is that, all things being equal, it is
easier for an English language speaker to publish in
an English language journal, which almost all SCI-
covered journals are. Of course, this is not so in the
most internationalised disciplines, but some effects
can be reasonably expected on the whole, which in-
troduces a bias.
Factor of bias 6: publications resulting from non-
public spending: So far, it has been assumed that all
scientific publications of a country are attributable to
public spending. This is not so, and we should dis-
count the publications linked to research financed by
non-public resources. It appears that more than 10%
of UK publications are produced by firms. Further-
more, it is well known that non-profit organisations
play a major role in financing basic research in the
UK and thus producing publications.5 The situation
is quite different in France where the non-public
related publications are significantly less. When
subtracting such publications from the total national,
the UK would loose more than France.
Assigning a differential impact of 10% on the value
of the productivity indicator for factors 3 and 4, and
an impact of 15% for factors 5 and 6, I find that the
value of the productivity indicator is 96 for the UK,
as compared to 100 for France, which means a dif-
ferential of 4% in favour of France.6
Has it been shown that France is more productive
in science than the UK? Certainly not. Beyond the
need to ground more firmly the corrections for bi-
ases 3 to 6, there is the need for a third round of cor-
rections to control for some additional factors of
bias, which may push the productivity differential in
either direction (factors 7 to 9):

Table 6. Effect of four factors of bias on the productivity indicator value (second round of corrections)

Country Corrected Factor of bias 3 Factor of bias 4 Factor of bias 5 Factor of bias 6 Value of indicator
indicator Unit cost for Weight of social Linguistic and Publications eliminating
eliminating salary and social sciences and cultural bias resulting from factors of bias
factors of bias security humanities non-public 1 to 6
1&2 spending

France 100 – – – – 100

a a a a
UK 164 –10% –10% –15% –15% 96
a
Note: estimates of the differential effect (bias) of the factors considered on the value of the productivity indicator

262 Science and Public Policy August 2001


The productivity of a research system
cannot be captured by a single
number, irrespective of how
sophisticated the indicator is: the
multiplicity of factors influencing the
indicator provides for an endless
debate over differences between
countries
• the teaching load of faculty and academic
researchers;7
• the perimeter of institutions included in academic
research spending;8 and
• the effects of the modes of financing of
universities.9
Other factors of bias can also be identified, each of
them bearing their own room for debate and contro-
versy. The point here is that the productivity of a
research system cannot be captured by a single
number, irrespective how sophisticated the indicator
may be.10 The multiplicity of factors influencing the
indicator while not related to productivity, provides
for an endless debate about differences between
countries.
A first lesson is that the indicator of Sir Robert,
besides being simple and available, does not demon-
strate anything regarding scientific productivity in
France and the UK. A second lesson is that the very
discussion of that productivity indicator has led to
engaging in the details of a comparative analysis of
the two research systems. The very act of critici-
sing the indicator has shown its potential to stimulate
in-depth analysis and debate. What are the implic a-
tions of this for the benchmarking of scientific
productivity?
Benchmarking between two extreme s
The implicit model behind the single productivity
indicator hypothesis is one where a subsystem called
‘scientific production’ can be identified and its re-
sources separated from all other flows, without any
feedback from the outside. Such a (linear) model is
flawed since it contradicts all we know about pro-
duction-of-knowledge processes and activities.
These are embedded in a context involving a vari-
ety of actors with differing aims and strategies,
interacting in networks of suppliers and users (Gib-
bons et al, 1995). In this conception, productivity
cannot be characterised by the ratio of one (among
many) output variable to one (among many) input
variable (ETAN Expert Working Group, 1999;
Georghiou, 1999).
Science and Public Policy August 2001
Sense and nonsense of S&T productivity indicators
Important questions are raised about the attribu-
tion of particular outputs to particular inputs, notably
whether a particular output would not have occurred
at all without the inputs that are presumed to have
caused it (additionality). Problems of irreversibility
and non-replicability are also involved (Guy, 1998).
There is no ‘productivity’ of research measurable as
such but a multi-actor dynamics involving knowl-
edge creation and diffusion.
Indeed, the fact that the productivity of a research
system cannot be captured by a single number di-
rectly derives from what we have learned over the
past 20 years about research activities: that they in-
volve systemic and interactive dynamics among a
variety of actors, linked to institutions, themselves
embedded in society. In other words, the nonsensical
character of the measurement of the productivity of
a research system is not linked to a technical diffi-
culty in finding the good number. It stems from the
fact there is no good number (Roqueplo, 1997).
A first extreme: impossibility of benchmarking
A possible implication of the model of the interac-
tive research system embedded in society is that no
simplification is possible and no quantitative com-
parison legitimate. The best we can do is provide
case studies (‘ad hoc stories’ illustrated by exam-
ples), which can only be put side by side. The
advantage of this view is that the case studies may
be meaningful to the actors and may provide valid
insights. Also it may lead to the development of a
conceptual framework, which would be based on a
real understanding of how things work.
The major problem of this view is that no real in-
teraction with other experiences is possible and no
transparency can be expected. The lack of a compre-
hensive conceptual framework coupled with the
specificity embedded in case studies implies that
benchmarking is virtually impossible.
The other extreme: oversimplified nonsense
In this view, research systems can be very much
simplified and represented in the form of a sequen-
tial mechanistic structure, so that they can be charac-
terised by a few simple, mostly input and output
indicators. Thus, their performance can be directly
derived and assessed by computing the ratio of
output to inputs, and they can be legitimately ranked
according to performance. Best practices, that is
those applied by those who are at the top of the
ranking system, should be applied elsewhere for
improvement.
The advantages of such a view are that it is sim-
ple, explicit, and easy to communicate to decision-
makers. In addition, it leads to the development of
sets of indicators, which may in any case be a useful
investment.
The biggest problem with this view is that it in-
volves a high risk of wrong recommendations,
263
Sense and nonsense of S&T productivity indicators
resulting in deterioration of performance and the
adoption of inferior practices. The risks stem from
the weak cognitive content of this approach, which
can only be part of top-down management. This lack
of content would lead to difficulties in enrolling the
relevant actors who would have difficulty position-
ing themselves meaningfully into the picture, and as
a result policies are likely to be unsuccessful.
Joint strategic evaluation process
Reconsidering the status of quantitative indicators
To arrive at a practical and methodologically sound
position, I suggest that the role of quantific ation
should be reconsidered, and the idea of a direct link-
age between indicators and productivity assessment
should be abandoned. Instead, I introduce the con-
cept of an informed and pluralistic debate in a wider
network, involving scientists and decision-makers. I
suggest that quantitative indicators are the starting
point for the discussion, with their raison d’être
being to be criticised in terms of their (limited) rele-
vance and (limited) comparability.
Indicators are considered thus, not as a final result
to be accepted, but as an entry point for debate. This
is an excellent way to enter an exercise of learning-
by-comparing, which is what benchmarking is
about. Criticism must be careful and positive, be-
cause the purpose is not to dismiss the legitimacy of
the exercise, but to help dig further for a better
understanding of the situation. Indicators will pro-
vide the cornerstone of the debates. There will be a
sequence of quantitative and qualitative analysis
phases, each one feeding from the previous and
into the next, in a virtuous circle of comparative
understanding.
Benchmarking as joint strategic monitoring exercise
To be relevant to public policy, benchmarkingmust be
considered as a process involving some aspects of
both extremes presented above. From the first, we
should keep the idea of analysis with sufficient depth
and due consideration of the socio-political and in-
stitutional context, and from the second we should
keep the idea of indicators useful for comparisons.
Thus, benchmarking must be viewed as a strategic
monitoring exercise resulting from the inter-
action and debate among experts from two countries
based on quantitative and qualitative evidence.
In this interpretation, benchmarking necessarily
involves both analytical work and direct interaction
among people. The experts are given the task of
criticising and interpreting jointly a set of indicators,
explicitly based on a common conceptual framework
and computed in a transparent way. The debate
among them is about the assumptions, the meaning
of the categories chosen to compute the indicators,
the bias as a result of specific features or
264
The proposed benchmarking is an
extended strategic monitoring process,
an exercise in joint evaluation and
assessment, a joint learning process
allowing international relations to
develop aimed at meaningful
comparison and understanding
mechanisms overlooked, very much in the spirit of
the discussion above.
In the course of such exchanges, deeper under-
standing of one’s own system emerges at the same
time as an understanding of the other systems. This
results in an assessment of threats, opportunities,
strengths and weaknesses, resulting in lessons and
experiences learned and possibly recommendations.
It also results in insights in alternative institutional
arrangements and incentive structures, ideas about
more adequate tools and procedures, a need for a
more appropriate description of the system, with
more appropriate concepts, and a need for better and
more relevant indicators.
Those results and needs should then feed into a
further stage of the benchmarking, which itself is a
multi-stage process. In this sense, benchmarking as
proposed here has much to do with an extended stra-
tegic monitoring process. It is an exercise in joint
strategic evaluation, joint assessment and a joint
learning process. Benchmarking thus provides an
opportunity for policy analysts, administrators and
decision-makers to develop international relations
aimed at meaningful comparison and understanding
through substantial interactions. To sum up, the fol-
lowing steps are suggested for implementing this
process:
Step 1: Development of a conceptual framework to
set up the categories, parameters and classific ations
needed to define a model of how things work.
Step 2: Selection and computation of internationally
comparable quantitative input, process and output
indicators.
Step 3: Benchmarking, with the help of national ex-
perts, which involves:
• characterising the structure of national S&T
production, based on the conceptual framework,
that is, relating the conceptual framework’s cat-
egories to the ‘real’ national innovation system,
its institutions and actors;
• interpreting and criticising the quantitative
indicators, accounting for national specificity of
various kinds;
Science and Public Policy August 2001
 Sense and nonsense of S&T productivity indicators

• complementing, when necessary, the quantitative
indicators by qualitative information.
Step 4: Identification of the key determinants of
S&T production effectiveness, as well as the
strengths/weaknesses of each national research and
innovation system.
Step 5: Relating these determinants (strengths/
weaknesses) to policy options in key areas of gov-
ernment policy (such as financing of public research,
prioritisation and evaluation of public research, de-
gree of autonomy of, and competition between, uni-
versities, forms of incentive programmes, and ele-
ments of the incentive structure), with a view to
making country-specific policy recommendations.
research system involves, may lead to the total dis-
missal of benchmarking as a tool for public policy.
There is room for benchmarking as a public policy
tool, provided that indicators are considered not as
results, but as entry points for debate, and that
benchmarking is understood as a multi-stage exer-
cise involving analysts, research actors and policy-
makers. To make sense, it should be seen as a joint
strategic evaluation, joint assessment and joint learn-
ing process.
Benchmarking thus provides an opportunity for
policy analysts, administrators and decision-makers
to develop international relations aimed at meaning-
ful comparison and understanding through substan-
tive interactions. In that sense, benchmarking can
indeed be a no-nonsense policy instrument.

A schematic representation of the five steps of the

proposed benchmarking processes is given in Figure Notes
1.
1. Scientific publications recorded in the Science Citation Index
(SCI) database built from about 3,500 scientific journals
Conclusions which are the most cited. In what follows, we will refer to
these scientific journals as ‘SCI journals’
2. The UK publishes 18.83 articles per million pounds sterling
Originating in the field of business management, and France 8.86.
3. OECD data: for France, CNRS is included in this total; each
benchmarking is under high risk of falling into the faculty member is counted for half ‘full-time research equiva-
trap of simplification nonsense when applied naively lent’
(or ideologically) to public policy. At the opposite 4. It could be argued that wage differentials should not be con-
sidered as a bias, but that it is in essence an element of
end of the spectrum, a rigid understanding of what a productivity. We rather address here the question of the effi-
ciency of the research system and thus the notion of the
productivity of the scientists as persons. The scientists’ wage
differential between the two countries is considered as a
1. bias, since we are concerned here with the quality of the
system and thus we must avoid mixing that with the question
conceptual of wages.
2. framework 5. This is particularly true of the medical research sector, with
the role of the Wellcome trust and of non-hospitals (see
background Hicks and Katz, 1997).
report: 6. I am well aware of the hypothetical character of our quantif i-
cation. The numbers are plausible estimates, but each one
3 can be discussed.
expert group benchmarking 7. In the OECD figures used in this analysis, the expenses from
seminar university personnel are considered to be 50% dedicated to
2a research activities. This figure of 50% is simply a statistical
national S&T
structure
standard, and the real situation may be completely different
presentation
discussion and in different institutions, disciplines, categories of personnel,
interpretation stage in career, the overall resulting figure being unknown
experts and possibly very significantly above or below the standard
50% figure.
8. In France, for example, it includes many vocational training
4 institutions which are administratively linked to the Ministry of
expert group benchmarking Higher Education, thus being added to the universities for
seminar budgetary figures. All administrative personnel are also
counted, which may include restaurants, cleaning, mainte-
nance. This may, or may not, be the case in the UK aca-
strengths and demic spending figures.
weaknesses 9. Is it certain that the financial resources coming from
students, from local authorities, from private firms, from
foundations are accounted for in the same way?
10. We can imagine all kinds of econometric indices, including
2b
5 very complicated ones, but that would not change anything.
indicators
computation expert group benchmarking
seminar
References
policy options
recommendations
ETAN Expert Working Group (1999), Options and Limits for
Assessing the Socio-Economic Impact of European RTD
Programmes , EUR 18884.
Figure 1. Proposed benchmarking process L Georghiou (1999), “Meta-evaluation: evaluation of evaluations”,

Science and Public Policy August 2001 265

Sense and nonsense of S&T productivity indicators
Special Issue on Proceedings of the Conference Science and
the Academic System in Transition — an International Expert
Meeting on Evaluation, Vienna, 3–5 July, Scientometrics, 45,
pages 523–530.
M Gibbons, C Limoges, H Nowotny, S Schwartzman, P Scott and
M Trow (1995), The New Production of Knowledge (Sage pub-
lications, London).
K Guy (1998), “Conceptual issues in RTD impact assessment”,
DGXII workshop measurement of RTD results/impacts,
266
Brussels, 28-29 May.
D Hicks and S Katz (1997), “The changing shape of British indus-
trial research”, STEEP special report no 6, SPRU, University
of Sussex.
R M May (1998), “The scientific investments of nations”, Science,
281, 3 July, pages 49–51.
P Roqueplo (1997), “Entre savoir et décision, l’expertise scienti-
fique”, collection ‘sciences en questions’ (INRA éditions,
Paris).
Science and Public Policy August 2001