BRIEF COMMUNICATION

What Is Behind the Curtain of the Leiden Ranking?

Rüdiger Mutz
Professorship for Social Psychology and Research on Higher Education, D-GESS, ETH Zurich, Muehlegasse
21, 8001 Zurich, Switzerland. E-mail: mutz@gess.ethz.ch

Hans-Dieter Daniel
Professorship for Social Psychology and Research on Higher Education, D-GESS, ETH Zurich, Muehlegasse
21, 8001 Zurich, Switzerland; Evaluation Office, University of Zurich, Muehlegasse 21, 8001 Zurich,
Switzerland. E-mail: daniel@gess.ethz.ch

Even with very well-documented rankings of universities,
it is difficult for an individual university to reconstruct its
position in the ranking. What is the reason behind
whether a university places higher or lower in the
ranking? Taking the example of ETH Zurich, the aim of
this communication is to reconstruct how the high posi-
tion of ETHZ (in Europe rank no. 1 in PP[top 10%]) in the
Centre for Science and Technology Studies (CWTS)
Leiden Ranking 2013 in the field “social sciences, arts
and humanities” came about. According to our analyses,
the bibliometric indicator values of a university depend
very strongly on weights that result in differing estimates
of both the total number of a university’s publications and
the number of publications with a citation impact in the
90th percentile, or PP(top 10%). In addition, we examine
the effect of weights at the level of individual publica-
tions. Based on the results, we offer recommendations
for improving the Leiden Ranking (for example, publica-
tion of sample calculations to increase transparency).
Introduction
Even when rankings of universities are very well docu-
mented, some uncertainty always remains with respect to
how the rankings actually came about, and this applies also
to the Leiden Ranking produced by the Centre for Science
and Technology Studies (CWTS) at Leiden University
(Waltman et al., 2012). According to the CWTS Leiden
Ranking website:
Received May 5, 2014; revised May 28, 2014; accepted June 5, 2014
© 2015 The Authors. Journal of the Association for Information Sciences
and Technology published by Wiley Periodicals, Inc. on behalf of
ASIS&T • Published online 29 April 2015 in Wiley Online Library
(wileyonlinelibrary.com). DOI: 10.1002/asi.23360
The CWTS Leiden Ranking 2013 measures the scientific per-
formance of 500 major universities worldwide. Using a
sophisticated set of bibliometric indicators, the ranking aims to
provide highly accurate measurements of the scientific impact
of universities . . . The CWTS Leiden Ranking 2013 is based on
Web of Science indexed publications from the period 2008–
2011 (http://www.leidenranking.com/ranking/2013).
To be able to draw valid conclusions based on rankings,
information on how the raw data have been weighted and
aggregated is extremely important. Our look behind the
curtain of the Leiden Ranking, which is at present the best
research ranking of universities worldwide based on a rig-
orous data analytic framework, can reveal central assump-
tions and decisions on the part of the CWTS that affect
results and that the user is not at all aware of when quickly
interpreting ranking results. But knowledge of those
assumptions and decisions is indispensable for correct inter-
pretation and use of the results. Criticisms (see next section)
aside.
Taking the example of ETH Zurich (ETHZ) in Switzer-
land and the Leiden Ranking main field of “social sciences,
arts, and humanities” (SOC_HU), we reconstruct how a rank
came about (citation impact, PP[top 10%]) also using the raw
data made available to us by the CWTS. ETHZ is “one of the
leading international universities for technology and the
natural sciences” (www.ethz.ch/en/the-eth-zurich.html),
which also has departments and research centers in SOC_HU
(e.g., Department of Humanities, Social and Political Sci-
ences; Competence Centre for the History of Knowledge).
In the field of SOC_HU, ETHZ ranks number one in
Europe in the Leiden Ranking 2013, with the highest per-
centage of PP(top 10%) publications (journal articles,
reviews). Of the total 358 publications of the ETHZ, 16.6%

JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY, 66(9):1950–1953, 2015
This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and
distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are
made.
FIG. 1. Screenshot of the EXCEL table (extract) of the 31 most-cited publications of the ETHZ (raw data) in the field of SOC_HU. The documents were
sorted in descending order by the total number of citations (CS). We did the fractional counting (fraction) shown here in column 1. [Color figure can be
viewed in the online issue, which is available at wileyonlinelibrary.com.]
are in the 10% most-cited publications in the field of
SOC_HU. This basically favorable result immediately raises
the question as to what disciplines and organizational units
in the field “social sciences, arts, and humanities” contrib-
uted to ETHZ’s top position in the ranking.
Organizational Units Versus Journal Sets
The main fields in the Leiden Ranking are defined based
on journal subject categories, and they should on no account
be equated with organizational units. Journal subject catego-
ries are journal sets1 of selected subfields. For the ETH
publications, a total of 54 subfields (that is, journal sets) are
assigned to the main field of SOC_HU. The list of subfields
includes, among others, Agricultural Economics & Policy,
Economics, Education & Educational Research, History &
Philosophy of Science, Information Science & Library
Science, Law, Management, Political Science, Sociology,
Transportation, Urban Studies, and Women’s Studies. With
1
The CWTS Leiden Ranking 2014 published on April 30th, 2014 differs
from the CWTS Leiden Ranking 2013 in one respect: “The fields have been
defined algorithmically in a unique bottom-up fashion based on millions of
citation relations between publications in the Web of Science data base.”
(CWTS Press release Leiden Ranking 2014). “828 fields are distinguished.
These fields are defined at the level of individual publications”
(www.leidenranking.com/methodology/indicators).
JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY—September 2015
this quite heterogeneous list of subfields, unambiguous
assignment of the publications and the resulting rank posi-
tion in the Leiden Ranking to a particular organizational unit
such as the Department of Humanities, Social and Political
Sciences is not possible. For example, the most-cited article
of ETHZ up to the end of 2012 in SOC_HU published in the
analysis period from 2008 to 2011 (Figure 1, row 2) is Engel,
Pagiola, and Wunder (2008). The first author’ affiliation,
however, is the Department of Environmental Systems
Science, and not the Department of Humanities, Social and
Political Sciences. The authors of the most-cited publica-
tions of the ETHZ (raw data) in the field of SOC_HU in
Figure 1 are affiliated with the Department of Environmental
Systems Science, just mentioned, and the Department of
Humanities, Social and Political Sciences, the Department of
Management, Technology and Economics, the Department
of Chemistry and Applied Biosciences and the Department
of Health Sciences and Technology, among others.
Raw Data Versus “Advanced Parameters”
Analysis
Here we take a closer look at the rank position of ETHZ
in SOC_HU for the indicator PP(top 10%):
• What publications and citations are considered in
SOC_HU?
1951
DOI: 10.1002/asi
• How is the total number of ETHZ publications in SOC_HU
determined?
• How is the number of ETHZ publications in the top 10%
calculated?
• What are the effects of weightings at the level of individual
publications?
What Publications and Citations Are Considered in
SOC_HU?
All publications (articles, reviews) classified as SOC_HU
in the time period 2008–2011/2012 were included in the
Leiden Ranking, that is, publications up to 2011 and cita-
tions up to 2012 (author self-citations are not included).
How Is the Total Number of ETHZ Publications in
SOC_HU Determined?
According to Figure 1, ETHZ published a total of 358
articles in the field of SOC_HU, of which 16.6% belong to
the 10% most-cited publications in their journal set.
However, when interpreting this finding it is important to
note any “advanced parameters” selected for the ranking
calculation. For instance, publications in special types of
journals (journals not published in English, journals without
strong international focus, trade journals, and popular jour-
nals) are excluded. When articles published in special types
of journals are included, the total number of articles pub-
lished by ETHZ in SOC_HU increases to 441, if all other
advanced parameters are selected.
But the advanced parameter with the greatest impact on
the calculated total number of articles is “calculate impact
indicators using fractional counting”:
The full counting method gives equal weight to all publications
of a university. The fractional counting method gives less
weight to collaborative publications than to non-collaborative
ones. For instance, if the address list of a publication contains
five addresses and two of these addresses belong to a particular
university, then the publication has a weight of 2 / 5 = 0.4
(www.leidenranking.com/methodology/indicators).
If this advanced parameter is not selected, but all other
advanced parameters are, the result for ETHZ is now 614
articles. And if the advanced parameter “exclude publica-
tions in special types of journals” is additionally deselected,
ETHZ has 730 publications. But that number does not equal
the actual number of ETHZ articles published in the main
field of SOC_HU, which is—according to the raw data—
854 publications. The difference between the two numbers
(730 vs. 854) is explained by the fact that some journals in
the Web of Science belong to multiple fields (i.e., journal
subject categories), and it may be that a journal belongs to
one or more SOC_HU subfields and to one or more science,
engineering, or life science fields.
Therefore, each publication has a weight that indicates
how strongly the journal in which the article was published
belongs to SOC_HU (CWTS, personal communication,
1952 JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY—September 2015
DOI: 10.1002/asi
2013). If, for example, a journal belongs to an SOC_HU
subfield and to three other, non-SOC_HU fields, the journal
has a weight of 0.25 instead of the 1.0 that the journal would
be given if it were assigned to the field of SOC_HU only.
The EXCEL table provided by the CWTS (see excerpt in
Figure 1) shows this in column H, “weight.” The sum of the
weights is 729.93 (∼730) publications. Thus, depending on
the advanced parameters selected, the number of publica-
tions varies from 358 (with the advanced parameters chosen
in the Leiden Ranking) to 854 (raw data set). By different
combinations of selection and deselection of advanced
parameters except for calculation of size-independent indi-
cators the position of the ETH Zurich does not change (first
position). This is, however, not the case for all universities in
the ranking (e.g., Imperial College London).
How Is the Number of ETHZ Publications in the Top
10% Calculated?
Weights also affect how citation impact is determined.
The value of the indicator PP(top 10%) for ETHZ varies in
the Leiden Ranking from 16.6% to 18.2%. The reason for
this is the following:
Most publications either do or do not belong to the top 10%
most highly cited of their publication year and field, but some
publications are partially considered to be 10% publications.
This is the case if a publication belongs to multiple fields and is
in the top 10% in some of these fields but not in others. It is also
the case if the number of citation of a publication equals exactly
the top 10% threshold (CWTS, personal communication, 2013;
Waltman & Schreiber, 2013).
To take this into account, “partly top 10%” publications
are given a weight below 1.0. PP(top 10%) is actually a
binary variable, with 1 for publications that belong to the top
10% and 0 for publications that do not belong to the top
10%.
Without any kind of weighting (PP(top 10%) > 0), based
on the raw data, a total of 223 of 854 publications of the
ETHZ are in the top 10% with regard to some reference
distribution (multiple fields); that is, 26.1% versus 16.6% in
the case of all selected “advanced parameters.” With the
advanced parameter “partly top 10%,” the result is 133 pub-
lications in the top 10% highly cited papers. Finally, if all
advanced parameters are selected (fractional counting,
excluding publications in special types of journals, size-
independent indicators), the number of publications in the
top 10% is only 60. The number of ETHZ publications in the
top 10% thus varies absolutely between 60 (with advanced
parameters) and 223 (raw data) and in percent between
16.6% (with advanced parameters) and 26.1% (raw data).
Similar calculations can also be performed for the other
citation impact measures (total citations/CS, field normal-
ized citations/NCS). It should be noted that fractional count-
ing was not only done for a single publication, but also for
all publications in reference sets for each field. Due to
fractional counting, for instance, the PP(top 10%) value for
each field is exactly 10%.
What Are the Effects of Weightings at the Level of
Individual Publications?
The effects of an advanced parameter analysis are seen
most tellingly at the level of the individual publication
(EXCEL table of raw data; see Figure 1). The most-cited
ETHZ article in the field of SOC_HU, Engel et al. (2008),
has a weight of 0.5. This weight results from the fact that the
journal Ecological Economics is evidently not only assigned
to the field of SOC_HU but also to a science subfield (e.g.,
Environmental Sciences & Ecology). In other words, this
highly cited ETHZ article in the field of SOC_HU is only
half credited. Using fractional counting of 0.33 (Figure 1,
column 1) this same article becomes less important again for
ETHZ’s ranking in the field of SOC_HU. The effect of the
selection of advanced parameters is even more drastic in the
case of Schneider, Holzer, and Hoffmann (2008), published
in the journal Energy Policy, which appears at the bottom
(row 32) of the table in Figure 1. This article was evidently
assigned to both the field of SOC_HU and two other fields,
so that it was given a weight of 0.33. In addition, it belongs
to the top 10% most-cited publications in only two of the
three fields. This explains the PP(top 10%) value of 0.67, or
2/3. In the total number of publications of ETHZ in the top
10% (N = 133), the Schneider et al. article is thus entered
with a value of only 0.67*0.33 = 0.22. If in addition the
fractional counting of 0.50 (see Figure 1 table, column 1)
was also selected, the resulting value would be even smaller.
Although the Leiden Ranking is one of the most presti-
gious university rankings worldwide, several things remain
unsatisfactory:
A subject/discipline comparison of universities, for example in
the field of SOC_HU, cannot be achieved using the Leiden
Ranking. The journal sets that are assigned to the field of
SOC_HU do not allow unambiguous assignment of subjects
and organizational units (c.f. Robinson-García & Calero-
Medina, 2014).
The user of the Leiden Ranking is not given sufficient
information on the nature of the raw data. The rounding up
of weighted values wrongly conveys the impression that the
values represent countable integers.
No sensitivity analyses are conducted that would deliver
information on what effects the different weightings have on
the ranking results. It would be helpful if the Leiden
Ranking website provided simple calculation examples
showing how a specific ranking is calculated based on the
JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY—September 2015
raw data. This would increase the transparency of the
ranking.
Also not very satisfactory is the methodological-
statistical approach. On the one hand, basic statistical con-
cepts are used, such as percentiles or means, but on the other
hand, pure mathematically based weightings are used. On
the one hand, these weights guarantee that the numbers, for
instance the top10% values, keep constant across various
fields, on the other hand, such weights might be difficult
in a statistical analysis (e.g., van den Boogart &
Tolosana-Delgado, 2013). What is lacking is an overarching
statistical model of the data of the sort that mathematicians
and statisticians have called for (Adler, Ewing, & Taylor,
2009; Goldstein, 2014).
Last but not least, we think that as a matter of urgency
more research should be conducted on the various weak-
nesses not only of the Leiden Ranking but also of interna-
tional university rankings in general (e.g., sensitivity
analysis, weighting, fractional counting, statistical model)
(Bornmann, Mutz, & Daniel, 2013).
References
Adler, R., Ewing, J., & Taylor, P. (2009). Citation statistics. A report from
the International Mathematical Union (IMU) in cooperation with the
International Council of Industrial and Applied Mathematics (ICIAM)
and the Institute of Mathematical Statistics (IMS). Statistical Science, 24,
1–14.
Bornmann, L., Mutz, R., & Daniel, H.-D. (2013). A multilevel-statistical
reformulation of citation-based university rankings: The Leiden Ranking
2011/2012. Journal of the American Society for Information Science and
Technology, 64, 1649–1658.
CWTS. (2013). Personal correspondence (emails 18.12./19.12.2013, M.
Neijssel, Leiden Ranking data).
Engel, S., Pagiola, S., & Wunder, S. (2008). Designing payments for
environmental services in theory and practice: An overview of the issues.
Ecological Economics, 65, 663–674.
Goldstein, H. (2014). Using league table rankings in public policy forma-
tion: Statistical issues. Annual Review of Statistics and Its Application, 1,
385–399.
Robinson-García, N., & Calero-Medina, C. (2014). What do university
rankings by fields rank? Exploring discrepancies between the organiza-
tional structure of universities and bibliometric classifications. Sciento-
metrics, 98, 1955–1970.
Schneider, M., Holzer, A., & Hoffmann, V.H. (2008). Understanding the
CDM’s contribution to technology transfer. Energy Policy, 36, 2930–
2938.
Van den Boogart, K.G., & Tolosana-Delgado, R. (2013). Analyzing com-
positional data with R. New York: Springer.
Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E.C.M., Tijssen,
R.J.W., van Eck, N.J. . . . (2012). The Leiden ranking 2011/2012: Data
collection, indicators, and interpretation. Journal of the American
Society for Information Science and Technology, 63, 2419–2432.
Waltman, L., & Schreiber, M. (2013). On the calculation of percentile-
based bibliometric indicators. Journal of the American Society for Infor-
mation Science and Technology, 64, 372–379.
1953
DOI: 10.1002/asi