Scientometrics

https://doi.org/10.1007/s11192-023-04798-z

Altmetrics‑based sleeping beauties: necessity or just

a supplement?

Jianhua Hou1 · Hao Li1 · Yang Zhang1 

Received: 16 June 2022 / Accepted: 20 July 2023

© Akadémiai Kiadó, Budapest, Hungary 2023

Abstract
With the rapid growth of social media, knowledge diffusion of scientific literature com-
prises not only citation trajectory but also dimension of societal impact. For special types
of knowledge diffusion, especially Sleeping Beauties in Science (SBs), it is essential but
not adequate to only use citation analysis to measure its trajectory. The impact and value of
scientific literature can be fully revealed only by integrating the diffusion trajectory of cita-
tion and social media. This study focuses on identifying the possible problems in research
on SBs and emphasizes the necessity of Altmetrics-based SB (A-SB) to solve them. Refer-
ring to the practice in our earlier study, we identified 7 A-SBs and 11 citation-based SBs
(C-SB). The comparative analysis of interdisciplinary and impact between A-SB and C-SB
revealed the likely defects in the traditional C-SB research on the measurement of impact
in knowledge diffusion, the mining of potential mechanism, the explanation of scientific
knowledge diffusion pathways and the discovery of some key scientific literature, and that
A-SB plays a vital role in these aspects. In contrast to C-SB, A-SB also offers the possibil-
ity to explore pathways for the diffusion of scientific knowledge. Furthermore, a typical
A-SB case analysis suggested that A-SB has a unique advantage in discovering some inno-
vative research with potential value ahead.

Keywords  Sleeping beauty · Altmetrics · Citation analysis · Knowledge diffusion

Jianhua Hou and Hao Li have contributed equally to the work.

* Yang Zhang
zhyang2@mail.sysu.edu.cn
Jianhua Hou
houjh5@mail.sysu.edu.cn
Hao Li
lihao86@mail2.sysu.edu.cn
1
School of Information Management, Sun Yat-Sen University, No.132, Waihuan East
Road, Guangzhou Higher Education Mega Center, Panyu District, Guangzhou 510006,
People’s Republic of China

13
Vol.:(0123456789)
 Scientometrics

Introduction

Traditionally, Sleeping Beauties in Science (SBs) has been considered a unique cita-
tion trajectory representing the Delayed Recognition phenomenon in science (Garfield,
1980), that is, the phenomenon that scientific literature was not cited for a prolonged
time after publication (Sleeping) and then suddenly received plentiful citations (Awak-
ened by Prince; van Raan, 2004). As it differs from the typical citation trajectory, it is
considered one of the Seven Myths in Bibliometrics (Glänzel, 2008), which fascinates
bibliometricians. Based on the quantitative standards proposed by van Raan (2004),
which made large-scale quantitative research on SBs feasible, the existing research
explains the special mechanism of SBs in the diffusion of knowledge and the necessity
of its existence from the standpoint of citation trajectory (referred to as Citation-based
SBs, C-SB; Braun et al., 2010).
However, with the rapid progress of social media, knowledge diffusion of scientific
literature involves not only citation trajectory but also the dimension of societal impact
(Thelwall et al., 2015). As social media, blogs, knowledge basis, and other platforms
that can be used to share and acquire scientific literature have become increasingly
vital channels for scientific knowledge exchange (Priem et  al., 2010), Altmetrics is
increasingly demonstrating strong vitality in measuring the societal impact of scien-
tific literature (Melero, 2015). The scientific communication behaviors (such as view,
save, discuss, and recommend) between scholars and the public on these platforms can
be captured in real-time, and the trajectory of impact can be formed in time, providing
the feasibility to comprehensively consider the academic impact and societal impact of
scientific literature in knowledge diffusion. However, their impact trajectories also suf-
fer from SBs (Hou & Yang, 2020). Therefore, it is necessary to consider both the SBs
phenomenon within the scientific community and involve sleeping beauty papers that
integrate the social dimension of knowledge diffusion (Gorry & Ragouet, 2016).
Existing study illustrated the existence value and awakening mechanism of SBs
from the viewpoint of comprehensive impact (Hou et al., 2020). However, for SBs, is
A-SB just a beneficial supplement or essential? In other words, can research on SBs
conducted just from the standpoint of citation trajectory restore the full picture of SBs
completely? Specifically, can the possible connections between some crucial features
of C-SB and the process of knowledge diffusion be examined fully without consider-
ing the trajectory of societal impact? In particular, interdisciplinarity, as an important
feature of SBs (Rinia, et al., 2001; Ke, et al., 2015; Du & Wu, 2018), is still relatively
obscurely understood in relation to SBs. Moreover, will the quantitative identification
of C-SB miss some chief literature with potential value? This study focuses on answer-
ing these questions and emphasizes the necessity of A-SB to solve them. The research
objectives of this paper are as follows:

(1) To demonstrate the necessity of A-SB for work related to SBs through comparative
analysis with C-SB.
(2) To analyze the applicability and accuracy of A-SB in identifying SBs in social media
environments through case analysis.
(3) To explore the advantages and contributions of A-SB in scientific knowledge diffusion
and potential knowledge discovery in social media environments.

13
Scientometrics

Literature review

Criteria for identifying C‑SB

High-quality identification criteria are the premise to discuss the key characteristics and
enlightenment of C-SB. As the citation number can measure the academic impact of sci-
entific literature to a certain extent (Merton, 1973), it is considered a crucial index for
tracing the trajectory of knowledge diffusion (Chen & Hicks, 2004). The curve created by
the change of the citation number over time after the publication of scientific literature is
called citation history (Redner, 2005) or citation trajectory (Hyett & Parker, 2009). Based
on the citation trajectory, the "three-indexes method" proposed by van Raan (2004) and
"beauty coefficient" proposed by Ke et al. (2015) are the most commonly used C-SB iden-
tification methods, representing subjective parameter method and no-parameter method,
respectively. Certainly, the enhancement of these two methods by scholars provides greater
convenience for empirical analysis of C-SB. For example, Costas et  al (2010) proposed
quartile-based criteria, which considers the diversity between different disciplines while
formulating parameters. Sun et  al. (2016) used the Gs index to measure the citation tra-
jectory variation, which improves the accuracy of distinguishing "sleeping beauties" from
normal articles. Du and Wu (2018) proposed the Bcp index, which substitutes yearly cita-
tions in "beauty coefficient" with yearly cumulative percentage of citations, to overcome
the overdependence of the no-parameter method on citation accumulation. More recently,
co-citation analysis has also been used to identify C-SB and its princes (Teixeira et  al.,
2017; Zhao et al., 2020; Zong et al., 2018), which still relies on subjective parameters. For
the empirical analysis, both the subjective parameter method and the no-parameter method
have advantages and disadvantages. The subjective parameter method needs to formulate a
fixed threshold value, which warrants a complete understanding of the differences between
different disciplines (Sun et  al., 2016). The no-parameter method sidesteps the negative
impact of subjective threshold, enabling easy identification of C-SB from large-scale
data. Nevertheless, in the empirical analysis of citation trajectory characteristics of C-SB,
the standard with parameters has unique advantages. Although methods with subjective
parameters do not accurately identify certain potentially valuable papers, identification
work using parameter-free methods was previously biased because it did not distinguish
between different disciplines and different points in the literature cycle (Gorry & Ragouet,
2016), which is clearly an advantage of methods with subjective parameters.

Key characteristics and enlightenment of C‑SB

By applying the above mentioned methods to the recognition and analysis of C-SB in dif-
ferent fields, researchers have supplemented the mechanism of knowledge diffusion and
provided a reference for science and technology policy. While the process of academic
development is an essential factor in the emergence of C-SB (Egghe et  al., 2011), some
excellent articles (Zhang & Ye, 2020) are presented as C-SB on their citation trajectory
(Li & Ye, 2014a, 2014b; Zhang & Ye, 2020). Of these, interdisciplinarity is a critical fac-
tor widely concerned by researchers. Previously, interdisciplinary research was considered
to be more susceptible to delayed recognition (Rinia et al., 2001). However, a large-scale
multidisciplinary citation structure revealed that the "citing" prince of C-SB often comes
from other disciplines, which is associated with the fact that C-SB mostly comes from

13
 Scientometrics

interdisciplinary journals (Ke et al., 2015). Nonetheless, another study reported that C-SB
and their prince are more likely to appear in the same field (Miura et al., 2021). Moreover,
a recent study indicated that C-SB has a unique advantage in identifying key innovations
and paradigm shifts in science (Kokol et al., 2020). In this process, self-citation could be
the prelude to the awakening of C-SB (Achouchi & Gorry, 2018), while lack of exces-
sive self-citation could be the cause of awakening (van Raan, 2021). Moreover, the social
emergencies-driven is a critical factor in the awakening of C-SB (Haghani & Varamini,
2021). Meanwhile, the analysis of the common characteristics of SBs brought unique and
crucial enlightenment to knowledge diffusion, knowledge innovation, scientific discovery,
scientific evaluation, and other fields. First, SBs can often reflect significant discoveries in
their fields or changes of a knowledge structure (Du & Wu, 2018; Ke et al., 2015). Sec-
ond, precise identification of SBs is conducive to shortening the scientific cognition cycle
and protecting crucial scientific discoveries (Guo & Yan, 2016). Third, recognition of SBs
affects scientific evaluation policies that focus on short-term citation windows and cumula-
tive effects (Hu & Rousseau, 2017). Overall, interdisciplinary, technology and application
attributes, and high quality are the important characteristics of SBs, and are also the clues
to explore the value of them (Du & Wu, 2018).
Further, some characteristics of C-SB outside the academic dimension have also been
discussed. After discovering that C-SB is technology-driven, van Raan (2015) further
explored their trajectory in the technical dimension and revealed that C-SB is often cited
by patents (van Raan, 2017), often helping C-SB being cited in advance (van Raan & Win-
nink, 2018). Meanwhile, it has been called for to consider the impact of informal commu-
nication on knowledge diffusion when analyzing C-SB (Gorry & Ragouet, 2016).

Measuring the societal impact of scientific literature based on Altmetrics

Although the "artificial pollution" of Altmetrics indexes has been extensively explored
(Roemer & Borchardt, 2015; Thelwall et al., 2013; Bornmann et al., 2019), the fact that the
impact reflected by Altmetrics indexes is inconsistent with the academic impact reflected
by citation seems to have reached a consensus (Michaela et al., 2021). There are ever more
calls to consider Altmetrics indexes a representation of the societal impact of scientific
research and measure the comprehensive impact of scientific research based on combin-
ing Altmetrics and citation (García-Villar, 2021; O’Connor et  al., 2017), rather than just
a supplement to citation (academic impact) (Das & Mishra, 2014). This is because some
scientific literature in a particular article type or subject area tends to have a wider societal
impact, which are overlooked owing to their low citation (Haustein et al., 2015a, 2015b;
O’Connor et al., 2017). In the societal dimension, the knowledge diffusion of scientific lit-
erature is affected by mention time and type of Twitter account (Zhang et al., 2018), differ-
ences of disciplinary (Holmberg & Thelwall, 2014), and early Web usage statistics (Wang
et al., 2019). Besides, scientific collaboration assists in improving the academic and social
impact of scientific literature (Wang et al., 2019; Zingg et al., 2020). All these conclusions
are based on the cumulative value of the social impact of scientific literature.
Further research showed that Altmetrics indexes could timely capture behaviors of
social media users, which is considered helpful in overcoming the lack of citation analysis
in societal impact measurement and early assessment for scientific literature (Mohammadi
et al., 2018; Roemer & Borchardt, 2015; Thelwall, 2018). Thus, some researchers started
focusing on the trajectory of societal impact. Relevant studies investigated the SBs phe-
nomenon on Twitter (Araujo, 2020) and identified the SBs based on social media with

13
Scientometrics

quantitative criteria (Hou & Yang, 2020). A previous work also quantitatively explored
SBs based on combing academic impact and societal impact of scientific literature (Hou
et al., 2020).
Based on the literature review above, it can be seen that Citation analysis has always
been almost the only means to elucidate the trajectory of knowledge diffusion and identify
SBs. With the emergence and rapid growth of social media, academicians became increas-
ingly aware of the crucial role of social media in measuring knowledge diffusion and
societal impact. Owing to issues, such as low coverage and easily manipulated, whether
the data of social media effectively reveals the societal impact of scientific literature has
always remained debatable (Butler et  al., 2017; Cheung, 2013; Crotty, 2017; Dinsmore
et al., 2014; Hammarfelt, 2014; Haustein et al., 2013; Thelwall et al., 2013). Nevertheless,
with the standardization of Altmetrics, its efficacy of real-time capture of societal atten-
tion of scientific literature is being recognized gradually, and gaps in the field remain to be
filled (Akshai et  al., 2017; García-Villar, 2021; Taberner, 2018). Besides, popular topics
and opinion articles usually gain higher attention than an original article on a specific topic
(Melero, 2015), which also enlightens scholars on how to better grasp the current social
trends. However, the current related research has an ample expansion space in, at least, the
following aspects:

(i) the identification criteria do not consider the societal dimension of knowledge dif-
fusion. The research on C-SB aims to address the problem that some key knowledge
cannot be diffused in time, but it is apparent that knowledge diffusion is affected by
societal factors like academic organization and public debate (Gorry & Ragouet,
2016). Some crucial indexes and even potential key articles might be omitted when
the related research is performed just based on the citation. The impact and value of
scientific literature can be fully revealed only by integrating the diffusion trajectory
of citation and social media.
(ii) More accurate methods should be used to quantify the characteristics of C-SB. The
existing research has hardly used quantitative indicators to examine the interdiscipli-
nary characteristics of SBs, and it is crude to calculate the interdisciplinarity of C-SB
only by their citation network in knowledge diffusion, while the indexes measured
on the basis of the knowledge basis (references) can more precisely reflect whether
an article has noticeable interdisciplinarity or not (Leydesdorff et al., 2019a, 2019b;
Stirling, 2007) and can also provide a way for quantitative analysis of the correlation
between interdisciplinarity and other characteristics of C-SB.
(iii) Most quantitative studies of C-SB lack detailed case analysis, which could lead to
the neglect of some crucial articles with potential value (Achouchi & Gorry, 2018),
making it challenging to reveal the characteristics and value of SBs completely and
comprehensively.

Thus, this study aims to reinforce the viewpoint that it is necessary to investigate the
diffusion pattern of scientific knowledge from the standpoint of combing academic impact
and societal impact (hereinafter comprehensive impact). Besides, interdisciplinarity and
impact, as one of the two variables most explored in SBs research, should also be included
in the scope of our discussion. Hence, we comparatively analyzed A-SB and C-SB from
these two aspects, further revealing the shortages of SBs research from the single perspec-
tive of citation. We then determine whether the A-SBs meet the traditional definition of
SBs based on the changing trends in academic and societal impact. Meanwhile, we use case
analysis for a typical A-SB to illustrate the unique advantages of A-SB in these aspects.

13
 Scientometrics

Data and method

Data

PLoS (http://​www.​ploso​ne.​org/) is one of the earliest platforms to advocate Article-level

metrics (ALM). It classifies and integrates the collected multisource data, which helps
to reflect the trajectory of academic and societal impact produced by an article (Lin &
Fenner, 2013a) and makes up for the limitation that citation "cannot fully reflect the
attention and scientific impact of a paper" (Lin & Fenner, 2013b). PLoS One is a com-
prehensive journal that was launched in 2006. Thus, considering our threshold setting
for A-SB (see "Measurement of the impact of articles" section), we took the annual dis-
tribution (from publication to 2020) of Cited (Ci), Viewed (Vi), Saved (Sa), Discussed
(Di), and Recommended (Re) of articles published in PLoS One between 2006 and
2011 (a total of 29,003 articles) as the analysis object. The data were obtained from the
PLoS One Open-Access Platform and the Core Collection database of Web of Science.
Table 1 provides the source and definition of each index.
The main reasons for choosing PLoS One as the data source are as follows: (i) PLoS.
org is convenient to trace the source platform of Altmetrics indicators and obtain the
time series of Altmetrics indicators. Historical data or time series data for social media
metrics were not available at Altmetric, Bookmetrix, Mendeley, Dimensions, and
PlumX to construct an impact trajectory for the paper. (ii) PLoS One has great advan-
tages in the time of publication and the number of papers, which is indispensable to
ensure the acquisition of rich and mature citation tracks (Li & Ye, 2014a, 2014b). (iii)
PLoS is a comprehensive journal, which differs greatly from traditional journals in pub-
lication mode. The papers included include biology and medicine, physics, chemistry,
social science, information and computer science and other disciplines. This provides a
multidisciplinary sample for us to analyze the interdisciplinary characteristics.
After collecting the samples and their indexes data, we first cleaned the data to
remove the relevant information of retraction, correction, letter, biological item, and
some editorial materials. Then, we removed some clearly wrong data (e.g., an article
obtained considerable Cited or Viewed a few years before publication, which is obvi-
ously not caused by pre-print) and duplicate data (e.g., if an article was repeatedly
mentioned by a user on Twitter and the content of the abstract remains unchanged, we
considered that this will pollute its social impact). Finally, there were 29,003 articles
and > 1.5 million valid data. We used Excel to classify, calculate, and form panel data
involving article, time, and multiple indexes as the target data of this study. Unlike ear-
lier Altmetrics studies, we selected panel data created by the annual distribution of each
index in each article as samples to completely reflect the evolution of the impact trajec-
tory of articles since their publication.
For Altmetrics related research, the coverage of indicators is an important criterion
for measuring the effectiveness and weight of Altmetrics indicators. Table 2 shows cov-
erage statistics for Viewed, Saved, Discussed, and Recommended.
Of note, only 322 articles (1%) obtained were recommended in the samples. As this
index is usually considered indexes with low coverage and poor applicability (Zahedi
et al., 2014; Priem et al., 2012), they were not suitable for evaluating the impact of liter-
ature. Nevertheless, this index is a record for qualitative evaluation of an article by peer
experts and can represent another dimension of both academic impact and social impact
different from the citation, which is rare for an article. To describe the impact trajectory

13
Scientometrics

Table 1  Definition and source of each index in articles of PLoS One categorized into cited, viewed, saved,
discussed, and recommended
Index Definition Data sources

Cited Number of times the literature was cited in other aca- Web of Science Core Col-
demic literature, i.e. citation counts lection
Viewed Sum of web views and downloads for online document PLoS One (http://​www.​
(HTML, PDF, XML) ploso​ne.​org/)
Saved Number of times online documents were saved in the Mendeley readership
document manager counted by Mendeley
(www.​mende​ley.​com)
Discussed Number of times articles have been shared, liked, Twitter, Facebook, Reddit,
commented on, and acquired in social tools such as Wikipedia
Twitter
Recommended Source data provided by PLOS Publishing Group Recommendations counted
through online recommendation channels and other by Faculty Opinions
platforms for obtaining formal recognition of PLOS
research papers

PLoS One integrates the data sources of each index, and the source of each index can be traced through the
PLoS Metrics interface. Under the index system of PLoS.org, Mendeley readership is named "Saved" (to
highlight the behavioral significance of the index). Therefore, in order to keep consistent with other indexes
in PLoS.org in terms of index name, we use the name "Saved" instead of Mendeley readership

of literature more completely, we did not remove indexes with low coverage, but only
considered their accessibility.
In general, Viewed represents the sum of the HTML page views, PDF downloads, and
XML downloads for a paper, Discussed represents the shares of a paper on Twitter and
Facebook, and Saved represents the number of times a paper was collected and followed
on social bookmarking platforms. Compared with Cited, although their coverage rate is
lower, they occur earlier, which can reflect the changing impact of a paper over time before
citations accrue, and also represent the different dimension of impact of a certain paper and
the academic interest of users. Thus, in combination with citations, using different types of
interactive data from from different platforms can effectively incorporate both academic
and social metrics (Lin & Fenner, 2013a). However, PLoS.org only provides a simple
count of various metrics, and does not provide a weighted aggregated metric like Altmet-
ric.com and PlumX.
Finally, considering the difference of numerical dimension between each index (Costas
et  al., 2015) and the time attribute of panel data, we uniformly standardized the data of
each index of each article in each year:
( )
( ) Xaj ti
Maj ti = ( ), (1)
Xaj ti

where a denotes article number, a = 1,2, …, 29,003; j denotes index number, j = 1,2, …, 5;
ti denotes time, which in this study is the i-th year after an article’s publication; Xaj ti
( )

denotes the mean value of Xaj ti with regard to both article and year; Maj ti is the stand-
( ) ( )

ardized data processed by Xaj ti .

( )

13
 Scientometrics

Table 2  Coverage of Altmetrics Index Number of papers with index Coverage rate

indexes in articles of PLoS One  values greater than 0 (N = 29,003)
(%)

Viewed 29,003 100

Saved 28,987 99.9
Discussed 4123 14.2
Recommended 321 1.1

Measurement of the impact of articles

We used the Analytic Hierarchy Process to assign weights for five indexes Cited, Viewed,
Saved, Discussed, and Recommended, and measured the comprehensive impact generated
in each year after the publication of an article (Altmetrics-based beauty index, Ab), that is,
the function value of community action of academic impact (IA) and societal impact (IS)
were as follows (Hou et al., 2020):
Ab = f(IA, IS)
How do we reflect the academic impact and societal impact of articles? In this study,
Cited (Ci) denotes the academic impact of literature; Viewed (Vi), Saved (Sa), Discussed
(Di), and Recommended (Re) reflect the societal impact of literature. Based on the synthe-
sis of these five indexes (Fig. 1), the annual distribution of Ab was described to form the
evolution trajectory of impact of an article in time series.
Thus, the dynamic change formula of the Ab Index for the i-th year since the publication
of a document was as follows:
Abi = Wtv ⋅ Vi + Wts ⋅ Si + Wtd ⋅ Di + Wtr ⋅ Ri + Wtc ⋅ Ci , (2)
where Wtv , Wts , Wtd , Wtr , and Wtc denote the corresponding weights of V, S, D, R, and C,
respectively; i denotes time, which in this study is the ith year after an article’s publication;
and Abi denotes the comprehensive impact of the ith year after an article’s publication. The
determination of weights Wtv , Wts , Wtd , Wtr , Wtc , and the specific process of Ab measure-
ment were as follows:
Compare the impact of n factors X1 , X2 , … , Xn on a factor (influence) at the previous
level, you can choose Xi and Xj from X1 , X2 , … , Xn to compare their contribution to influ-
ence (Or importance). Assign values to Xi/Xj per the following "1–9 scale".

Xij Implication

1 The influence of Xi is equivalent with that of Xj

3 The influence of Xi is slightly stronger than that of Xj
5 The influence of Xi is stronger than that of Xj
7 The influence of Xi is obviously stronger than that of Xj
9 The influence of Xi is absolutely stronger than that of Xj
2,4,6,8 The ratio of influence of Xi and Xj is between the above two adjacent levels
1,1/2,…,1/9 The ratio of influence of Xi and Xj is the inverse of the ratio above

13
Scientometrics

Fig. 1  Hierarchy model of Ab 

Building the model matrix:

Xij Vi Sa Re Di Cd

Vi 1 1/2 1/7 1/7 1/9

Sa 2 1 1/5 1/5 1/8
Re 7 5 1 1 1/5
Di 7 5 1 1 1/5
Cd 9 8 5 5 1

By evaluating the maximum of its eigenvalues is λmax = 5.2837, and the eigenvector at

the maximum eigenvalue was as follows:

(0.0515 0.0770 0.2827 0.2827 0.9919)T

By performing the consistency test on the results, CI = 𝜆 max
n−1
−n
= 5.2837−5
5−1
= 0.0709 , the
consistency ratio was CR = RI = 0.063 < 0.1 (RI is the random consistency test; when n = 5,
CI

RI = 1.12). Hence, the matrix passed the consistency test. By conducting the normalization
processing on the eigenvector, the weights were obtained, that is, Wtv = 0.0321, Wts = 0.048,
Wtd = 0.176, Wtr = 0.176, and Wtc = 0.5679.
The setting of weight of each indicator is based on the results of existing empirical stud-
ies, theoretical studies and consultations with five experts in the field of Altmetrics. For
example, compared with other Altmetrics indexes, Saved as Mendeley readership played
a significant role in measuring the impact of scientific papers. In practice, Mendeley
readership is considered to be a good metric with high coverage for assessing scholarly
outcomes(Haustein et al., 2015a, 2015b; Li et al., 2012; Zahedi et al., 2014;). Theoretically,
Altmetrics can be divided into the perception level, the social media level and the applica-
tion level (Qiu & Yu, 2015). Mendeley readership reflects users’ subscription and collec-
tion behavior of scientific papers (Wang et al., 2014), which is a relatively high-level index
in social media layer. From the perspective of impact generation mechanism, it has a higher
contribution than discussed, viewed and recommended (Qiu & Yu, 2015), so we assign
Saved a weight higher than Discussed, Viewed, and Recommended.

13
 Scientometrics

Hence, article a in the Altmetrics-based beauty index of ti can be expressed as follows:

Aba (ti ) = 0.0321Va (ti ) + 0.048Sa (ti ) + 0.176Da (ti ) + 0.176Ra (ti ) + 0.5679Ca (ti ). (3)

Method for identifying A‑SB

In previous work, the trajectory of the comprehensive impact of literature were measured
in months. Nevertheless, considering the data availability and to ensure the comparability
of A-SB and C-SB, we recalculated in years in this study. Thus, the related concepts and
parameters involved in this study merit explanation.

Threshold setting

The setting of parameters and threshold directly influences the impact of recognition of
SBs, and it is also a critical method to judge the state of an article. Thus, considering the
influence of journals, disciplines, and other differences on the impact of literature (Gorry
& Ragouet, 2016; Mcallister et al., 1983; Ravenscroft, 2017; Waltman, 2016), and to mini-
mize the negative impact of subjectivity and arbitrariness (Li & Ye, 2016), we took the
average value of Ab of all samples in each year ( Ab ) as the threshold to determine whether
an A-SB awakens or not.

Definition of concepts

We refer to the SBs created by the combination of citation index and social media indexes
as Altmetrics-based Sleeping Beauty (A-SB), for the following main reasons:
Firstly, epistemologically we have proposed the concept of A-SB based on the definition
of Altmetrics proposed by NISO. In the relevant expressions, Altmetrics is considered to
be a broad term integrates the collection of multiple digital indicators related to scholarly
work, that is, "it leaves open the possibility of the complementary use of these conven-
tional metrics, including for purposes of gauging scholarly impact" (NISO, 2016). So in a
broader sense, Altmetrics can include citation metrics.
Secondly, practices in established studies have inspired our proposed concept. On the
one hand, although the wording may lead to confusion, for a comprehensive measure of
the impact of scientific papers, citation metrics are still considered in existing Altmetrics-
related practices (Qin & Gao, 2019; Zhai et al, 2020; Zhang L et al., 2020). On the other
hand, given that existing studies focus on different dimensions (e.g. patent and pure social
media indexes) (Hou & Yang, 2019, 2020), we were concerned that this might lead readers
to over-understand the boundaries of our core concept and undermine the weight of Alt-
metrics in it (Altmetrics accounted for a significant proportion of our weighting scheme).
van Raan (2004) clearly defined the change of the trajectory of C-SB with three indexes,
that is, length of the sleep (s), depth of sleep (Cs), and awake intensity (Cw). This method
would help us precisely investigate the characteristics of the trajectory of SBs. Based on
this method and our research object, we judged the different stages of an article through
the persistence of different states. Hence, we describe the important concepts involved as
follows:

13
Scientometrics

Sleep

When the Ab value of an article in a certain year is not higher than Ab∕ 2, this article is
considered in a state of sleep in that year. If an article remains in the state of sleep after
publication, and this state lasts for ≥ 4 consecutive years, it is considered in the sleep stage.
The duration of the sleep stage is called sleep time (Ts).

Awakening

When the Ab value of an article in a certain year is not lower than Ab , this article is con-
sidered in a state of awakening in that year. After the sleep stage, when an article remains
in the state of awakening for the first time, we judge whether it enters the awakening stage;
only when this state lasts for ≥ 4 consecutive years, it is considered in the awakening stage
(this is the vital standard to identify A-SB). The duration of the awakening stage is called
awakening time (Tw).

Depth of sleep and awakening intensity

To describe the characteristics at different stages, we used two variables, average of Ab

( Ab(j−i) ) and standard deviation of Ab ( σ ) of each stage to explain the depth of sleep and
awakening intensity of an A-SB. When in the sleep stage, Ab(j−i) ≤ Ab∕2 , the smaller the
Ab(j−i) and σ , the stronger the depth of sleep of this A-SB. On the contrary, when in the
awakening stage, Ab(j−i) ≥ Ab , the larger the Ab(j−i) and σ , the stronger the awakening
intensity of this A-SB.
Overall, this method carefully considers the standardization and weighting of citations
and Altmetrics, and uses the average comprehensive impact of all papers over the entire
lifecycle as the basic threshold for identifying A-SB, rather than a completely subjective
threshold. This has advantages over existing C-SB studies and is conducive to reasonably
demonstrating the contribution of Altmetrics in A-SB recognition work. The significance
of this method lies in its alignment with the NISO concept of supplementing Altmetric to
measure the overall impact of research outputs. From this point of view, the introduction
of Altmetrics is undoubtedly conducive to the identification of sleeping beauty in the new
environment of knowledge diffusion.

Method for identifying C‑SB

We identified C-SB based on the analysis framework proposed by van Raan (2004). Spe-
cifically, we refer to van Raan(2015) and have already adjusted the three subjective thresh-
olds in van Raan’s (2004) method in the empirical study (relatively looser than van Raan’s
(2004) threshold). Make it identify as many C-SBs as possible and guarantee the accuracy,
so that we can make an adequate comparative analysis between A-SBs and C-SBs. The
thresholds are set as follows: sleeping periods (Ts) >  = 5, sleep depth (Cs) <  = 2, awakening
periods (Ta) = 4, and awakening intensity (Ca) > 20. We selected this method for the follow-
ing reasons:

13
 Scientometrics

(1) The subjective parameter method proposed by van Raan (2004) is more suitable for
our research problem. Although the recognition of SBs should avoid the subjective
threshold (Li & Ye, 2016), the stage division of the citation trajectory of SBs, whether
it is the no-parameter method or the subjective parameter method, still depends on
the three-index method designed by van Raan (2004). Therefore, the method of van
Raan (2004) can clearly distinguish the sleeping and awakening of SBs, which is more
conducive to comparing the trajectory characteristics between A-SB and C-SB. In
addition, the accuracy of subjective threshold methods can be improved by modifying
thresholds within existing analytical frameworks (Gorry & Ragouet, 2016).
(2) The identification results of SBs based on the method proposed by van Raan (2004) are
more conducive to the subsequent empirical analysis. It should be considered whether
the SBs identified by these methods really meet the definition of "Sleeping Beauty of
science". The subjective parameter method can help us accurately identify the type of
citation curve that conforms to the definition of "Sleeping Beauty", so as to discuss
the subsequent impact or value of this type of articles, while the no-parameter method
represented by the Beauty coefficient (Ke et al., 2015) has some inaccuracies in the
case empirical analysis of SBs (Gorry & Ragouet, 2016).
(3) In our sample set, the recognition result of method proposed by van Raan (2004) seems
to be more accurate than that of no-parametric method. Before conducting empirical
analysis, we have selected the improved model of the Beauty coefficient, that is, the
Bcp index (Du & Wu, 2018), and compared the recognition results calculated by it
with the recognition results calculated by van Raan (2004) as follows. It was found that
although the C-SB identified by Bcp presented the characteristics of SBs on the curve,
most of them had obtained higher citation absolute value in the so-called sleep stage.
See Table A.1 for the results. The results identified by Bcp are the top 11 articles in
the Bcp index.

Method for measuring the interdisciplinarity of A‑SB and C‑SB

The essence of interdisciplinarity is the integration of knowledge basis from different dis-
ciplines, and the interdisciplinarity measured by the discipline and field difference of refer-
ences denotes the integration diversity of the knowledge basis by scientific research (Stir-
ling, 2007). Thus, to clearly characterize the difference of interdisciplinarity between A-SB
and C-SB, we selected the ture diversity (TD) index (Zhang et al., 2016) and DIV index
(Leydesdorff et al., 2019a, 2019b) with high discrimination to measure the interdisciplinar-
ity of articles, as well as made statistics on the variety, balance, and disparity (Leydesdorff
et al., 2019a, 2019b) of a single article.
Specifically, TD is an improved index of the traditional Rao-Stirling diversity. Its advan-
tage is that it improves the comparability of interdisciplinarity among papers in the set. Its
formula is as follows:
1
TD = ∑ , (4)
1 − (1 − i,j Sij Pi Pj )

where Pi and Pj respectively represent the proportion of papers belonging to discipline

category i and discipline category j to papers belonging to all discipline categories in the
references. Sij is the similarity between discipline categories i and j, calculated through
cosine similarity using the co-citation matrix of Web of Science Subject Categories.
1 − i,j Sij Pi Pj is the Rao-Stirling diversity (Zhang et al., 2016).
∑

13
Scientometrics

DIV proposed by Leydesdorff et  al., (2019a, 2019b) operationalizes "Variety", "Bal-
ance," and "Disparity" independently and then combines them, which can better reflect the
difference between calculation results. Leydesdorff et  al. has provided a software on his
website, which can conveniently calculate the values of TD, DIV and other indexes.1 Both
indicators measure the diversity of references in an article according to the category of
Web of Science Subject Categories to which the journal of the reference belongs.
Based on the above mentioned threshold and concepts, we identified and empirically
analyzed A-SB. Figure 2 shows the specific process.

Results

Measurement of Ab

We calculated the comprehensive impact of all samples in each year (Ab) and obtained
the average value of the comprehensive impact of articles in each year ( Ab ≅ 1 and
Ab∕2 ≅ 0.5 ). To examine the efficacy of Ab in measuring the comprehensive impact of
articles, we first conducted Kolmogorov–Smirnov test (Table 3) on the Ab value of articles
and the five two-level indexes. The P values of each index were < 0.05, that is, they did not
conform to the normal distribution. In addition, there are differences in the dimensions
between the secondary indicators, indicating that it is necessary for us to carry out stand-
ardization work before calculating Ab.
Hence, we used the Spearman correlation coefficient to analyze the correlation between
each index (Fig.  3). A significant correlation was found between those indexes. Among
them, the Ab value of articles revealed a strong positive correlation with Cited, a moderate
positive correlation with Saved, Viewed, and Discussed, while a weak positive correlation
with Recommended, showing that Ab exerted a good integration effect on various indexes
and could reflect the academic impact and societal impact of articles simultaneously. From
the correlation among the two-level indexes, Cited had a significant moderate positive cor-
relation with Viewed and Saved, but a significant weak positive correlation with Discussed
and Recommended. The multicollinearity test (Table  4) suggested that the VIF value of
each index was 2.045, and no serious multicollinearity problem exists. This demonstrated
that each index was not only related but also could reflect the different dimensions of
impact of articles, verifying the efficacy of the hierarchical structure model of comprehen-
sive impact of articles (Ab; Fig. 1).
Based on this result and the flow of Fig. 2, we unfolded the identification and empirical
research of A-SB.

Identification of A‑SB and C‑SB

According to "Method for identifying A-SB" section, we identified 7 A-SB. Concurrently,

and used the method proposed by van Raan (2004) for reference to identify 11 C-SB (Table
A.2). From the recognition results, no duplicate article was found between A-SB and C-SB,
and the number of A-SB was less than that of C-SB (A-SB accounted for 0.02% of the total
samples and C-SB for 0.04%).

1
  https://​leyde​sdorff.​github.​io/​diver​sity_​measu​rement/

13
 Scientometrics

Fig. 2  The identification and empirical analysis process of A-SBs, which is also the research framework
of this study. The third column shows the process of identifying A-SB using the average of comprehensive
impact as the threshold. Among them, Ts refers to the sleep time, Tw refers to the awakening time, Ab refers
to the average comprehensive influence, and Ab(j−i) refers to the average value of Ab during that time period

Table 3  Descriptive statistics and one-sample Kolmogorov − Smirnov test of each indicator

Ab-total Cited-total Viewed-total Saved-total Discussed-total Recommended-
total

N 29,003 29,003 29,003 29,003 29,003 29,003

Normal parameters
Mean 11.01 46.89 6427.43 88.96 2.96 0.01
SD 47.48 70.84 10,288.25 114.99 59.99 0.13
Most extreme differences
Absolute 0.409 0.256 0.316 0.238 0.480 0.528
Positive 0.352 0.209 0.265 0.207 0.429 0.528
Negative − 0.409 − 0.256 − 0.316 − 0.238 − 0.480 − 0.461
Kolmogorov– 69.707 43.634 53.780 40.546 81.803 89.929
Smirnov Z
Asymp. Sig. .000 .000 .000 .000 .000 .000
(2-tailed)

13
Scientometrics

Fig. 3  Correlation between
each index. Ab means the total
Ab value; Ci denotes the total
Cited value; Vi denotes the total
Viewed value; Sa means the
total Saved value; Di means the
total Discussed value; and Re
means the total Recommended
value. Red, positive correlation;
blue, negative correlation. Color
depth indicates the value of the
coefficient: the darker the color,
the stronger the correlation. The
value at the center of each "round
dot" represents the P value of
each correlation coefficient.
(Color figure online)

Table 4  Collinearity diagnostics Indicators VIF

Ci 2.357
Vi 2.224
Di 1.655
Sa 2.923
Re 1.064
Mean 2.045

From the trajectory curve of articles, A-SB demonstrated an obvious mutation process
of "sleep–awakening" in the trajectory of comprehensive impact, and C-SB exhibited an
obvious mutation process of "sleep-awakening" on the trajectory of citation, which aligns
with the curve characteristics of SBs. However, as the three indexes used to identify C-SB
are subjective thresholds (on average, they are cited more than five times a year in the
awakening stage; van Raan, 2015), C-SB fluctuates briefly during sleep.
From the difference between the citation trajectory and the comprehensive impact tra-
jectory, significant differences were found between the citation trajectory and the compre-
hensive impact trajectory of 3 articles in A-SB, and there were 2 articles in C-SB. Further
analysis of the curve revealed that 11 C-SB failed to become A-SB because they gained
significant societal impact in their sleep stage (such as Montemurro, 2011), or failed to
obtain continuous and sufficient Ab value after the sleep stage. Based on the comprehen-
sive impact, A-SB was cited more after awakening than before, and their "sleep–awaken-
ing" processes did not neglect academic impact or societal impact, but the result of the
joint action of the two.

13
 Scientometrics

Table 5  Interdisciplinarity of A-SB and C-SB founded in PLOS One 

Type Interdisciplinarity TD DIV Variety Disparity Balance (GINI)

All Mean 6.296 0.0099 29.186 0.902 0.9996406

95% Confidence interval for mean
Lower bound 6.271 0.0098 29.05 0.902 0.9995730
Upper bound 6.321 0.0101 29.32 0.903 0.9997082
A-SB Mean 6.384 0.0069 24.429 0.925 0.9997293
95% Confidence interval for mean
Lower bound 4.653 0.0018 15.121 0.910 0.9996337
Upper bound 8.114 0.1197 33.736 0.941 0.9998248
C-SB Mean 6.423 0.0105 28.909 0.901 0.9996631
95% Confidence interval for mean
Lower bound 5.092 0.0042 20.473 0.877 0.9995602
Upper bound 7.755 0.0168 37.345 0.924 0.9997660

Comparative analysis of A‑SB and C‑SB

The impact trajectory of an article can reflect the diffusion process of the knowledge it
carries, but it is hard to reflex the internal mechanism of this formation process. Thus,
based on the temporal logic from knowledge basis to knowledge diffusion of article, we
compared and analyzed the interdisciplinarity and impact (including academic impact and
comprehensive impact) of A-SB and C-SB.

Interdisciplinarity comparative analysis between A‑SB and C‑SB

We used the interdisciplinarity measurement tool (Leydesdorff et  al., 2019a, 2019b) to
measure the interdisciplinarity of all samples (Table  5). To ensure the result discrimina-
tion, Variety takes the amount of disciplines involved in an article instead of dividing it by
the total amount of disciplines involved in all samples.
The average results of relevant indexes showed that the interdisciplinarity of C-SB was
more significant than that of A-SB. This is despite the disparity of C-SB being usually
lower than that of A-SB, and C-SB involves more categories than A-SB. Nevertheless,
based on the TD index, only 3 A-SB and 6 C-SB were higher than the average level of all
samples. From the DIV index, only 2 A-SB and 4 C-SB were higher than the average level
of all samples. Hence, from the standpoint of the knowledge basis, only a few A-SB and
C-SB were more interdisciplinary than the average level of all samples.
Under this difference, does the interdisciplinarity of A-SB and C-SB correlate with their
"sleep–awakening" characteristics? The Spearman correlation analysis (Table 6) revealed
no correlation between interdisciplinarity and the "three indexes" of C-SB. However, a sig-
nificant positive correlation was found between DIV index and Ab of sleeping of A-SB.
After refining the combination elements of the DIV index, we found a significant high cor-
relation between variety and Ab in the sleep stage. Besides, disparity significantly nega-
tively correlated with 𝜎 of sleep.

13
Scientometrics

Table 6  Correlation between interdisciplinarity and characteristics of SBs

Indicator TD DIV GINI Variety DISP

A-SB
 Ab of Sleep Correlation coefficient .107 .857* − .679 .857* .071
Sig. (2-tailed) .819 .014 .094 .014 .879
 Ab of Awakening Correlation coefficient − .071 − .071 − .071 − .071 .500
Sig. (2-tailed) .879 .879 .879 .879 .253
 𝜎 of Sleep Correlation coefficient − .429 .179 .000 .179 − .893**
Sig. (2-tailed) .337 .702 1.000 .702 .007
 𝜎 of Awakening Correlation coefficient − .214 − .357 .143 − .357 .214
Sig. (2-tailed) .645 .432 .760 .432 .645
Ts Correlation coefficient .408 − .612 .408 − .612 .408
Sig. (2-tailed) .363 .144 .363 .144 .363
Ta Correlation coefficient − .055 .037 − .128 .037 .275
Sig. (2-tailed) .907 .938 .784 .938 .550
C-SB
 Cs Correlation coefficient − .395 − .055 .023 .028 − .083
Sig. (2-tailed) .229 .872 .947 .936 .809
 Ca Correlation coefficient − .212 − .345 .400 − .399 − .193
Sig. (2-tailed) .532 .299 .223 .225 .569
 Ts Correlation coefficient .404 .231 − .173 .174 .231
Sig. (2-tailed) .218 .494 .611 .610 .494

Comparative analysis of the impact of A‑SB and C‑SB

Based on the citation analysis, SBs is usually considered as "high-quality research that
receives high attention after awakening." However, on the premise of a similar publica-
tion year, the 5% trimmed mean of the cumulative value of citation and Ab of A-SB
was higher than that of C-SB, as well as higher than the average level of all samples
(Table 7). The mean cumulative value of citations of C-SB and Ab were lower than the
average of all samples. Currently, C-SB’s academic impact and comprehensive impact
are at a low level, although this does not represent their final performance. At present,
the identified A-SB perform better than C-SB in academic impact and comprehensive
impact.
To further verify whether the identified A-SBs satisfy the definition of SBs, we ana-
lyzed the trajectory changes of A-SBs before and after awakening. Table A.3 shows that
in the awakening stage, 7 A-SBs had been more cited and saved than in the sleeping
stage, and 7 A-SBs had an increasing trend in saved and 6 A-SBs had an increasing
trend in cited. Although Ying E’s (2008) cited showed a downward trend since awak-
ening, it still had more cited than in sleeping stage and had an upward trend in saved.
However, all 7 A-SBs’ viewed showed a crest around the time of awakening and then
went into a downward trend. For Ponseti (2007), Proulx (2008) and Szczurko et  al.
(2007), they were all discussed in the year of awakening, and their discussed index had
a wave in the subsequent awakening phase.

13
 Scientometrics

Table 7  Comparison between Type Article Cited Ab

A-SB and C-SB on academic
impact and comprehensive
A-SB Szczurko (2007) 20 54.4919
impact
Xu (2010) 100 14.0367
Li (2009) 117 16.8159
Proulx (2008) 51 10.0022
Ying (2008) 81 11.7834
Moron (2008) 104 14.9761
Ponseti (2007) 28 24.5665
5% Trimmed mean 71.913 19.6984
C-SB Masia (2009) 36 5.8255
Freeman (2010) 33 5.4400
Pitcher (2010) 31 10.5464
Hernandez (2011) 32 4.9695
Mar (2011) 45 6.7829
Montemurro (2011) 32 10.4134
Si (2011) 29 4.3352
Toivari (2011) 29 4.2284
Corbett (2011) 36 5.2932
Wang (2011) 37 5.1696
Knoernschild (2011) 32 4.9536
5% Trimmed mean 33.46 6.0436
All 5% Trimmed mean 38.38 6.5512

Analysis of a typical A‑SB case: Szczurko et al. (2007)

The distribution of indicators in Szczurko et al. (2007)

Citation trajectory only represents the evolution process of academic impact of litera-
ture over time, but might not reflect the situation of the literature themselves. Thus, to
further examine the A-SB value, we selected an A-SB with the most noticeable char-
acteristics of "sleep–awakening" (the largest difference between the depth of sleep and
awakening intensity) as a typical case for in-depth mining. Table 8 and Fig. 4 provide
basic information and the time distribution of each index of this article.
From the standpoint of citation, Szczurko et al. (2007) does not conform to the defi-
nition of C-SB by van Raan (2004)—its Cw does not reach the threshold proposed by
van Raan (2004), and the accumulation of Cited is only 20 times (up to 2020). How-
ever, from the comprehensive impact perspective, the cumulative Ab value of this article
reached 54.49, which is of a high level.
Then, we analyzed the trajectory of comprehensive impact of this article. As shown
in Fig. 4, before awakening, the values of other indexes of this A-SB were low, except
for Viewed. In particular, during the sleep stage, this article was never discussed. Dur-
ing the awakening stage, Viewed of this article exhibited an apparent downward trend,
while the values of other indexes increased markedly (Fig. 5). Among them, obtaining a

13
Scientometrics

Ab

15

10

0
2007 2009 2011 2013 2015 2017 2019

Viewed Discussed Saved Cited

4
3000 30
20 3
2000 20 2
10
1000 10 1
0 0 0 0
2007 2009 2011 2013 2015 2017 2019 2007 2009 2011 2013 2015 2017 2019 2007 2009 2011 2013 2015 2017 2019 2007 2009 2011 2013 2015 2017 2019

Fig. 4  The distribution of various indicators with each passing year of Szczurko et al. (2007), including Ab
and its two-level indexes: viewed, discussed, saved, and cited. Of note, 2012 is the first year of its awaken-
ing; this article was not recommended from 2007 to 2020

large amount of Discussed explains why this article entered the awakening stage lasting
for 8 years.

Interpretation of the behavior behind the indicators

According to the trajectory of comprehensive impact and the distribution characteristics

of each secondary index, Szczurko et  al. (2007) has the obvious characteristics of SBs,
which meets the identification criteria of A-SB in "Method for identifying A-SB" section.
However, whether this case conforms to the formation and development logic of SBs quali-
tatively still needs further interpretation of the behavioral characteristics behind its indica-
tors. Considering that there is a certain time lag in cited compared with Altmetrics, we
interpreted the behavioral characteristics behind the indicators of Szczurko et  al. (2007)
from the two trajectories of academic impact and societal impact.

Interpretation of the trajectory of academic impact  Given that information from cit-

ing papers is an important tool for interpreting the academic impact trajectory of the
case, we used CitNetExplorer software to identify whether there were significant factors
contributing to Szczurko et al. (2007) awakening by visualizing its citation network. In
Fig. 6, it is clear that Szczurko et al. (2007) obtained two self-citations in 2008 and 2010
before the first year of awakening (2012). Among them, the self-citation paper in 2008
had four co-citation relationships with Szczurko et al. (2007). At the same time, there
were three other citations that were indirectly related to this node. It can be seen that a
self-citation before awakening has a certain influence on Szczurko et al. (2007) to obtain
more citations in the future. However, self-citation is not the only factor Szczurko et al.
(2007) enters the awakening phase. In the awakening stage, there were 13 Cited, 10 of
which directly cited Szczurko et al. (2007).
We performed an understanding reading of the content of the 20 citing articles (Table
A.4) to understand the citation motivation behind the indicator. From the topic of the cit-
ing papers, the research questions or research designs of the 20 citing papers were some-
what similar to those of Szczurko et al. (2007). For example, they focused on naturopathic
care, treatment of chronic low back pain and other research issues, and experimental design

13
 Scientometrics

Table 8  Basic information of Szczurko et  al. (2007), including author, title, indexes of depth of sleep,
indexes of awakening intensity, Ts and Tw
Author Title Ab of Ab of awak- 𝜎 of 𝜎 of awak- Ts Tw
sleeping ening sleeping ening

Szczurko Naturopathic care for chronic low 0.234 6.602 0.109 4.833 5 8
et al. back pain: a randomized trial
(20077)

methods such as randomized trial and whole systems research. In terms of thematic trends,
Szczurko et al. (2007) focused on clinical science and health science paradigms, and has
not yet achieved interdisciplinary knowledge diffusion. From the motivation of the citing
papers, the citation motivation of the 20 citing articles were all positive or neutral citations,
including affirming the research contribution of Szczurko et al. (2007), using the research
results or design of Szczurko et al. (2007) to support their views, and using the research
data of Szczurko et al. (2007) as samples. Among them, two citing papers clearly believe
that Szczurko et al. (2007) is an innovation or potentially disruptive innovation.

Interpretation of  the  trajectory of  societal impact  Since Viewed and Saved do not
provide user information, we only explored the specific behavior of Discussed. By 2020,
Szczurko et al. (2007) had been discussed 76 times, 55 of which originated from Twitter
and involved 30 Twitter users. 20 originated from public wall posts on Facebook, which
involved 11 Facebook users. 1 from Wikipedia.
We interpreted the user identity and the content of the text of indicators (Annex 1,
Table A.5). We found that users related to Szczurko et al. (2007) topic (individuals or
institutions that have noted naturopathic physicians and other relevant information in
their names or profiles) tended to post the results of the paper for popular science pur-
poses, to support their ideas in discussions, or to advocate for their profession. Or point
to a link to the paper, sharing information related to Szczurko et al. (2007). At the same
time, we noted that during the awakening stage, four users highlighted Szczurko et al.
(2007) innovative value to the field. So far, the majority of users participating in the dis-
cussion are highly relevant to the subject area in Szczurko et al. (2007).
To further understand the joint action of academic impact and societal impact of
A-SB in the process of knowledge diffusion, we must analyze it in combination with
its specific content and interpretation. To the best of our knowledge, Szczurko et  al.
(2007) is an innovative research on applying natural therapy in the treatment of chronic
low back pain, which established that "natural care provided significantly greater out-
comes than physiotherapy advice for chronic low back pain" through a randomized con-
trolled trial. Although this research won the attention of many users of journal web-
sites in the early stage, it is regarded as "demonstrate promise and innovation, despite a
lack of large-scale funding and methodologic challenges" (Ali & Katz, 2015). Hence, it
obtained little citations, and then Viewed exhibited a downward trend.
However, 5 years after its publication, owing to the innovation of this research, it was
widely mentioned and forwarded by researchers and the public on the social media plat-
form. Indeed, on the social media platform, the first person mentioned after its sleep is
a practitioner in the field of natural therapy, and his tweet was a simple comment on this
study. During the 8  years of the awakening stage, researchers, institutions, practition-
ers, and the public, who paid attention to the field of natural therapy, participated in the

13
Scientometrics

18

16

14

12

10

0
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019

Viewed Discussed Saved Cited Boundary of Awakening

Fig. 5  In the sleep stage and awakening stage, the proportion of secondary indicators of annual AB value of
Szczurko et al. (2007)

Fig. 6  Citation network of Szczurko et  al. (2007) over 2007–2020. Each point represents a paper, and
is labeled by the last name of the first author. The curved lines represent citation relations, and the cited
paper is always located above the citing paper. Among them, blue point indicates A-SB, i.e. Szczurko et al.
(2007). Purple points indicate self-citation, which means that one of the authors of the literature represented
by the node contributed to the writing of the A-SB. (Color figure online)

promotion and debate of this research. They generally recognized the innovative value of
the research and its significance as a milestone in related fields, and used it as a source
of knowledge for science popularization, communication, publicity and other activities.
Meanwhile, Viewed of this article exhibited a short growth (3  years), while Saved and
Cited exhibited a sustained and significant upward trend.

13
 Scientometrics

Discussion

It has become the consensus of the academic community that it should not be limited to
academic impact, but should also attempt to measure the societal impact, and that the latter
should be further weighted (Bornmann, 2013; Stern, 2016). In this context, integrating Alt-
metrics with citation analysis to characterise academic and societal impact is increasingly
important, as it takes into account the value of the paper in academic research and social
communication, and can refine the measurement of the comprehensive impact of scientific
papers (Wu & Zhao, 2016; Qin & Gao, 2019; Zhai et al., 2020; Zhang et al., 2020). Mean-
while, the result of the descriptive statistics of citation index and Altmetrics index shows
that the societal impact diffusion represented by Altmetrics has become an indispensable
process in the knowledge diffusion of most papers. Therefore, it is necessary to measure
the diffusion trajectory of the comprehensive impact and to identify from it this particular
trajectory of SBs. However, a purely quantitative analysis may lack a deeper understand-
ing of the ’messages’ behind those particular trajectories identified, i.e. there is a need to
explore the value of scientific papers through case studies and narrative analysis, which
would help to present a true picture of the comprehensive impact of scientific papers.
From the above epistemology, the core significance of introducing Altmetrics index
and designing corresponding identification methods lies in making the analysis framework
of traditional SBs adapt to the rapidly developing science communication background of
social media. Scientific knowledge is diffused not only within the scientific community,
but also through social media, knowledge base and other emerging media. To identify SBs
only based on citation index is not only inconsistent with the law of scientific development
and scientific knowledge diffusion, but also unfair to those scientific papers with unique
performance of impact trajectory on social media. Therefore, from the perspective of meth-
odology, A-SB considering alternative metrics is more conducive to identifying "real SBs"
in the context of social media than C-SB considering a single citation.
Based on a larger-sample dataset, this study demonstrates the feasibility of combining
Altmetrics indexes and citation to measure the trajectory of the comprehensive impact
of articles through quantitative analysis and case analysis, and change the measurement
units of impact trajectory for comparative analysis between A-SB and C-SB. For identi-
fication of C-SB, we selected a lower subjective threshold to identify as many C-SBs as
possible, while ensuring the accuracy of recognition (citation trajectory conforms to Van
Raan’s basic definition of C-SB), so as to facilitate the comparison between A-SB and
C-SB. Therefore, the sample is representative to some extent. The findings reveal a correla-
tion between Ab and its two-level indexes, as well as a correlation between each secondary
index, and these variables do not exist a serious multicollinearity problem. Thus, Ab inte-
grating academic impact and societal impact is an effective way to measure the trajectory
of the comprehensive impact of literature in knowledge diffusion. However, is it crucial
to identify SBs in this way as opposed to a single perspective of citation? In other words,
what can we learn from the identification and analysis of A-SB?
The results reflect that obvious differences exist between A-SB and C-SB. First, the con-
cern of some researchers is indeed true, that is, identifying SBs from a single viewpoint of
citation seemingly ignores the diffusion process in the social dimension of scientific lit-
erature (Gorry & Ragouet, 2016). We observed that the recognition results of A-SB and
C-SB do not coincide at all, which is due to the different impact reflected by Ab and Cited.
Specifically, C-SB is not A-SB because they gained excessive societal impact during sleep,
or that C-SB failed to attain sustained and sufficient societal impact after sleep, showing

13
Scientometrics

that the potential value of C-SB with "low academic impact and high societal impact" dur-
ing sleep might have been early concerned or recognized by some scholars and the public,
rather than awakening by "citing prince." Moreover, the C-SB that failed to awaken from
the standpoint of comprehensive impact, although they have been discovered by the aca-
demic community, are rarely perceived and understood in society. This may be related to
topics and audiences of the scientific knowledge carried by C-SB, e.g. some basic science
papers are simply incomprehensible to the general public due to their technical characteris-
tic, but they are fundamental for the academic community. However, in the long term, soci-
etal impact has become an increasingly important dimension in the evaluation of scientific
research, and the academic community should work to promote the impact of potentially
important scientific knowledge on dimensions beyond the academic community (Born-
mann, 2013, Parker & Van Teijlingen, 2012). In this perspective, their value has not been
explored fully.
Second, the comparison of interdisciplinarity between A-SB and C-SB demonstrates
that considering SBs only from the single perspective of citation might overlook some
crucial characteristics of knowledge diffusion. Based on their reference records, the inter-
disciplinarity of C-SB is more significant than that of A-SB; however, only a few A-SB
and C-SB are higher than the average level of all samples. It is suggested that although
C-SB is considered to tend to be interdisciplinary works for they are often awakened by
prince in another discipline (Ke et al., 2015), they might not be obvious interdisciplinary
research from the knowledge basis perspective. In addition, the variety and disparity of
knowledge basis highly correlate with the indexes of the sleeping stage of A-SB, but not
with the "three indexes" of C-SB. The knowledge basis of SBs exerts a potential impact on
the knowledge diffusion process integrating academic and societal dimensions, which is
overlooked from the single perspective of citation.
There was another interesting finding in the results. Several important characteristics
of SBs have been excavated in previous studies, and the interdisciplinary characteristics
of SBs have attracted the most attention of researchers. However, the quantitative result
of interdisciplinarity of C-SB shows that in the sample set, they do not have particularly
prominent interdisciplinary characteristics. There is still room to expand the understanding
of the interdisciplinary characteristics of SBs. This result motivates us not only to focus on
whether there is an intrinsic relationship between interdisciplinarity and their awakening,
but also to re-examine the interdisciplinarity of SBs, which is often overlooked in existing
studies of C-SB.
Thirdly, compared with C-SB, A-SB is more sensitive to articles with high comprehen-
sive impact and potential value. Although the citation trajectories of 7 A-SB articles do not
fulfill the criteria of van Raan (2004), from the comprehensive impact perspective, A-SB
is cited more after awakening than before. This demonstrates that the awakening of A-SB
not only reverberates on social media but also is a result of the joint action of academic
impact and societal impact. Another evidence is that the 5% trimmed mean of A-SB in the
cumulative value of citation and Ab is higher than that of C-SB and higher than the average
level of all samples, while the 5% censored mean value of C-SB in the cumulative value of
citation and Ab is lower than the average level of all samples.
In contrast to C-SB, A-SB also offers the possibility to explore pathways for the dif-
fusion of scientific knowledge. Despite the gradual ’decline’ of viewed, A-SBs perform
better in the awakening stage of cited, saved and discussed than in the sleeping stage. On
the one hand, this suggests that the awakening of A-SBs is the result of a sustained com-
bination of academic and societal impact, rather than a temporary preference for them by
certain groups. The identified A-SBs satisfy the basic definition of SBs. On the other hand,

13
 Scientometrics

the difference in trends in the three-level indicators before and after the awakening stage
may suggest a ’pass-through’ relationship (Qiu & Yu 2015) between academic and societal
impact. For the identified A-SBs, they, in line with the average paper, receive a large num-
ber of views at the outset. Whether or not these views translate into actual citation behavior
is a function of the success of that latent knowledge being awakened, and a measure of
whether or not an undiscovered knowledge is actually valuable. From this point of view,
A-SB reflects the dynamical process by which different levels of metrics are transformed,
and thus has an advantage over C-SB in resolving the awakening mechanism of SBs.
Finally, some articles that do not conform to C-SB in citation trajectory might encoun-
ter delayed recognition in the social impact dimension and can reflect crucial innovations.
A-SB has unique advantages over C-SB in finding such key literature ahead of time, and
can contribute to the understanding of knowledge diffusion in digital environment, espe-
cially in social media platforms. To further validate this issue, we analyzed a typical A-SB
case. As debatable innovative research in alternative medicine, this article might be less
quoted because of the particularity of its field (Thelwall, 2021), while it tends to be exten-
sively concerned by academics and the public on social media platforms (Mohammadi
et al., 2020). From the trajectory of comprehensive impact, this article has not been rec-
ognized widely in the academic circle or concerned by users of social media platforms
at first. However, owing to its pioneering work in this field, the article was promoted by
researchers in the same field on social media platforms 5 years after its publication. Subse-
quently, the users involved were concerned about its significance, and some still believed it
was contentious. After awakening, this study also attracted the attention of academia, who
have been viewed, saved, and cited to be accumulated as their knowledge basis. Owing to
the low accretion of citation or their cited does not reach the threshold of awakening, such
potential and valuable articles that had attracted more attention on social media (Haustein
et al., 2015a, 2015b) often do not attract the attention of bibliometricians.
The results of case analysis also show that although this typical A-SB’s citation litera-
ture collection and the audience on social media are highly similar to its own subject, in
terms of content, it has some characteristics of SBs that have been discovered. First, in
addition to direct citations, the authors’ self-citation also contributed to the accumulation
of citations in the awakening stage of the A-SB case. This is similar to the results of El
Aichouchi and Gorry (2018) ’s qualitative case analysis of a typical C-SB, it suggests that
we should pay attention to constancy and continuity of research, which is a favorable factor
to promote the early detection of those potentially valuable innovations. Second, in the aca-
demic community and social media activities, scholars generally recognized the innovative
value of this research and its landmark significance in related fields, cited it as a methodol-
ogy or viewpoint support, and used it as a knowledge source for science popularization,
communication, publicity and other activities.
Overall, the process of knowledge diffusion of scientific literature might be much more
complex than we think. Only paying attention to the academic impact of literature could
miss the social dimension of knowledge diffusion, which would lead us to disregard some
significant mechanisms of SBs in knowledge diffusion and some valuable innovative
but potential research. Hence, we believe that it is necessary to explore SBs and even all
knowledge diffusion based on the comprehensive impact of literature.

13
Scientometrics

Conclusions

To further illustrate the necessity of considering the comprehensive impact of literature in

the related research of knowledge diffusion, based on a larger sample dataset, the compara-
tive analysis of interdisciplinary and impact between A-SB and C-SB reveal the possible
defects in the traditional C-SB research on the measurement of impact in knowledge dif-
fusion, the mining of potential mechanism, the explanation of scientific knowledge diffu-
sion pathways and the discovery of some key scientific literature, and that A-SB plays a
vital role in these aspects. In contrast to C-SB, A-SB also offers the possibility to explore
pathways for the diffusion of scientific knowledge. Moreover, the case analysis of a typical
A-SB indicates that A-SB has a unique advantage in discovering some innovative research
with potential value ahead, which tends to get extensive social attention and citations after
their long-term sleep but are usually overlooked because their citation accumulation is low
or the citation number does not reach the awakening threshold needed by C-SB. Based on
the definition of SBs, the identification method of A-SB comprehensively considers the
diffusion trajectory of academic impact and society impact of articles. Its advantage is to
capture the societal impact of papers ignored in methods for C-SB, which is essential to
further reveal the character and value of SBs. With the advent of the paradigm revolu-
tion in measuring the impact of scientific papers (Bornmann, 2012), the societal impact is
increasingly valued. Therefore, identifying A-SB is not only a beneficial supplement, but
also a necessary practice in the new environment to the problems associated with SBs. We
believe that this study can provide some new insights into the existing research on knowl-
edge diffusion and formulation of science and technology policy. The contributions of this
study are as follows:

(1) This study demonstrates the necessity of A-SB for work related to SBs. The com-
parative analysis of interdisciplinary and impact between A-SB and C-SB reveals the
possible defects in the traditional C-SB research on the measurement of impact in
knowledge diffusion, the mining of potential mechanism, the explanation of scientific
knowledge diffusion pathways and the discovery of some key scientific literature, and
that A-SB plays a vital role in these aspects.
(2) The findings establish that identifying A-SB is more suitable for identifying some
innovative research with potential value in advance, especially in social media plat-
forms. The societal attention and citations of such scientific literature demonstrate a
significant and continuous increase after long-term sleep, but they are usually over-
looked because the citation accretion is low or the citation number does not reach the
awakening threshold needed by C-SB.
(3) This study enriches the perspective of exploring the awakening mechanism of SBs.
The difference in trends in the three-level indicators before and after the awakening
stage may suggest a ’pass-through’ relationship, and this dynamical process of trans-
formation of each indicators provides a deeper perspective to reveal the pathway of
scientific knowledge diffusion and potential knowledge discovery in the social media
environment.

Nevertheless, this study has some limitations worth acknowledging. First, the pro-
cess of knowledge diffusion of scientific literature is multidimensional. This study pro-
vides some insights into the dimension of the societal impact of diffusion but does not
comprehensively consider the policy dimension and patent dimension. Furthermore, the

13
 Scientometrics

’pass-through’ relationship between societal and academic impact remains unclear, and
the resolution of this complex issue is linked to a deeper understanding of the forma-
tion mechanisms of A-SBs, e.g. whether a scientist’s social media behavior triggers a
subsequent influx of citations that leads to the awakening of A-SBs, which can only
be resolved through a systematic study of the system of scientific knowledge diffusion
in new contexts. Besides, there are also factors to consider that may have influenced
Altmetrics, including the expansion of the Internet itself, the rise of social networks
and the culture of immediacy. Second, considering the space limitation, and in order to
highlight our above research problems, we selected the most representative dimension
of SBs for investigation. This study only discusses the influence of interdisciplinarity
of the knowledge basis of SBs, which aims to illustrate the differences and advantages
of A-SB over C-SB in mining knowledge diffusion mechanism, but apparently many
other influencing factors still warrant comparison, such as technology orientation. In
addition, the interdisciplinarity of SBs is a complicated issue that requires consider-
ing relevant indicators in different dimensions of interdisciplinarity. In particular, a pre-
liminary investigation has been made on the disciplines of A-SB and C-SB identified
(Fig. A.1), but due to the space, the discipline distribution of A-SB has not been further
analyzed. And that is crucial to finding out if there are topics in which A-SB was being
more relevant. Finally, although we carefully selected the data source of indexes, similar
to the existing Altmetrics studies, the data reliability and stability have not been recog-
nized fully. With the continuous improvement of Altmetrics, this limitation needs to be
solved. To complement the puzzle of knowledge diffusion, we will focus on investigat-
ing the differences between different types of curves from the viewpoint of the com-
prehensive impact in future studies, and the possible internal relationship between the
interdisciplinarity and knowledge diffusion of articles.
Supplementary Information The online version contains supplementary material available at https://​doi.​
org/​10.​1007/​s11192-​023-​04798-z.

Acknowledgements This research was supported by The National Social Science Fund of China(Grant
No. 20BTQ085), and The Soft Science Project of Science and Technology Program of Guangdong
Province(CN), 2019B101001024.

Declarations 
Conflict of interest  The authors have declared that no competing interests exist.

References
Araujo, R. F. (2020). Communities of attention networks: introducing qualitative and conversa-
tional perspectives for altmetrics[J]. Scientometrics, 124(3), 1793–1809. https://​doi.​org/​10.​1007/​
s11192-​020-​03566-7
Ali, A., & Katz, D. (2015). Disease prevention and health promotion: How integrative medicine fits. Ameri-
can Journal of Preventive Medicine., 49, S230–S240. https://​doi.​org/​10.​1016/j.​amepre.​2015.​07.​019
Akshai, D., Baheti, A. D., & Bhargava, P. (2017). Altmetrics: a measure of social attention toward scientific
research. Current Problems in Diagnostic Radiology, 46, 391–392. https://​doi.​org/​10.​1067/j.​cprad​iol.​
2017.​06.​005
Bornmann, L. (2012). Measuring the societal impact of research. EMBO Reports, 13(8), 673–676.
Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey.
Journal of the American Society for Information Science and Technology, 64(2), 217–233.
Bornmann, L. (2015). Usefulness of Altmetrics for measuring the broader impact of research: A case study
using data from plos and f1000prime. Aslib Journal of Information Management, 67(3), 305–319.

13
Scientometrics

Bornmann, L., Haunschild, R., & Adams, J. (2019). Do altmetrics assess societal impact in a comparable
way to case studies? An empirical test of the convergent validity of altmetrics based on data from the
UK research excellence framework (REF). Journal of Informetrics., 13, 325–340. https://​doi.​org/​10.​
1016/j.​joi.​2019.​01.​008
Braun, T., Glänzel, W., & Schubert, A. (2010). On Sleeping Beauties, Princes and other tales of citation
distributions. Research Evaluation, 19(3), 195–202.
Butler, J. S., Kaye, I. D., Sebastian, A. S., et al. (2017). The evolution of current research impact metrics:
From bibliometrics to altmetrics? Clinical Spine Surgery, 30, 226–228. https://​doi.​org/​10.​1097/​BSD.​
00000​00000​00053
Chen, C., & Hicks, D. (2004). Tracing knowledge diffusion. Scientometrics, 59(2), 199–211. https://​doi.​org/​
10.​1023/B:​SCIE.​00000​18528.​59913.​48
Cheung, M. (2013). Altmetrics: Too soon for use in assessment. Nature, 494, 176. https://​doi.​org/​10.​1038/​
49417​6d
Costas, R., van Leeuwen, T. N., & van Raan, A. F. J. (2010). Is scientific literature subject to a “sell-by-
date”? a general methodology to analyze the “durability” of scientific documents. Journal of the Amer-
ican Society for Information Science & Technology, 61(2), 329–339.
Costas, R., Zahedi, Z., & Wouters, P. (2015). The thematic orientation of publications mentioned on social
media: Large-scale disciplinary comparison of social media metrics with citations. Aslib Journal of
Information Management, 67(3), 260–288.
Crotty, D. (2017). Altmetrics. European Heart Journal, 38, 2647–2648. https://​doi.​org/​10.​1093/​eurhe​artj/​
ehx447
Das, A. K., & Mishra, S. (2014). Genesis of altmetrics or article-level metrics for measuring effificacy of
scholarly communications: Current perspectives. Journal of Scientometric Research, 3(2), 82–92.
Dinsmore, A., Allen, L., & Dolby, K. (2014). Alternative perspectives on impact:the potential of ALMs and
Altmetrics to inform funders about research impact. PLOS Biology, 12, 1002003. https://​doi.​org/​10.​
1371/​journ​al.​pbio.​10020​03
Du, J., & Wu, Y. S. (2018). A parameter-free index for identifying under-cited sleeping beauties in science.
Scientometrics, 116(2), 959–971.
Du, J., & Wu, Y. S. (2018). Sleeping beauties in science: Key characteristics, early identification clues and
science policy implications. Studies in Science of Science, 11, 1938–1945. https://​doi.​org/​10.​16192/j.​
cnki.​1003-​2053.​2018.​11.​003
Egghe, L., Guns, R., & Rousseau, R. (2011). Thoughts on uncitedness: Nobel laureates and Fields medal-
ists as case studies. Journal of the American Society for Information Science and Technology, 62(8),
1637–1644.
El Aichouchi, A., & Gorry, P. (2018). Delayed recognition of Judah Folkman’s hypothesis on tumor angio-
genesis: When a Prince awakens a Sleeping Beauty by self-citation. Scientometrics, 116(1), 385–399.
Garfield, E. (1980). Premature discovery or delayed recognition-why? Current Contents, 21, 5–10.
García-Villar, C. (2021). A critical review on altmetrics: Can we measure the social impact factor? Insights
into Imaging, 12(1), 92. https://​doi.​org/​10.​1186/​s13244-​021-​01033-2
Glänzel, W. (2008). Seven Myths in Bibliometrics. About facts and fiction in quantitative science studies.
In: Kretschmer, H. & Havemann, F. (eds.): Proceedings of WIS 2008, Fourth International Conference
on Webometrics, Informetrics and Scientometrics & Ninth COLLNET Meeting, (pp. 1–9).
Gorry, P., & Ragouet, P. (2016). “Sleeping beauty” and her restless sleep: Charles Dotter and the birth of
interventional radiology. Scientometrics, 107, 773–784.
Guo, F., & Yan, X. Y. (2016). Research review of sleeping beauty in science. Library Development, 05,
40–45.
Haghani, M., & Varamini, P. (2021). Temporal evolution, most influential studies and sleeping beauties of
the coronavirus literature. Scientometrics, 126(6), 1–46.
Hammarfelt, B. (2014). Using altmetrics for assessing research impact in the humanities. Scientometrics,
101(2), 1419–1430.
Haustein, S., Peters, I., Sugimoto, C. R., et al. (2013). Tweeting biomedicine: An analysis of tweets and cita-
tions in the biomedical literature. Journal of the Association for Information Science and Technology,
65(4), 656–669.
Haustein, S., Costas, R., & Larivière, V. (2015a). Characterizing social media metrics of scholarly papers:
The effect of document properties and collaboration patterns. PLoS ONE, 10(3), e0120495.
Haustein S, Bowman T, & Costas R. (2015b). Interpreting "altmetrics": Viewing acts on social media
through the lens of citation and social theories. Theories of Informetrics: A Festschrift in Honor of
Blaise Cronin.
Holmberg, K., & Thelwall, M. (2014). Disciplinary differences in Twitter scholarly communication. Scien-
tometrics, 101(2), 1027–1042.

13
 Scientometrics

Hou, J., & Yang, X. (2019). Patent sleeping beauties: Evolutionary trajectories and identification methods.
Scientometrics, 120(1), 187–215.
Hou, J., & Yang, X. (2020). Social media-based sleeping beauties: Defining, identifying and features. Jour-
nal of Informetrics, 14(2), 101012.
Hou, J., Li, H., Zhang, Y., & Gao, J. P. (2020). The evolutionary characteristics of sleeping beauties from
the perspectives of Altmetrics. Journal of the China Society for Scientific and Technical Information,
40(9), 934–952.
Hu, X., & Rousseau, R. (2017). Nobel Prize winners 2016: Igniting or sparking foundational publications?
Scientometrics, 110, 1053–1063. https://​doi.​org/​10.​1007/​s11192-​016-​2205-x
Hyett, M., & Parker, G. (2009). Can the highly cited psychiatric paper be predicted early? Australian and
New Zealand Journal of Psychiatry, 43(2), 173–176.
Ke, Q., Ferrara, E., Radicchi, F., & Flammini, A. (2015). Defining and identifying sleeping beauties in sci-
ence. Proceedings of the National Academy of Sciences of the United States of America, 112(24), 7426.
Kokol, P., Vosner, H. B., & Jernej, Z. (2020). Did Sleeping Papers in nursing research miss their target audi-
ence. Scientometrics, 122, 1243–1248. https://​doi.​org/​10.​1007/​s11192-​019-​03323-5
Li, J., & Ye, F. Y. (2014a). The phenomenon of all-elements-sleeping-beauties in scientific literature. Scien-
tometrics, 92(3), 795–799.
Li, J., & Ye, F. Y. (2014b). A probe into the citation patterns of high-quality and high-impact publications.
Malaysian Journal of Library & Information Science, 19(2), 17–33.
Li, J., & Ye, F. Y. (2016). Distinguishing sleeping beauties in science. Scientometrics, 108(2), 1–8.
Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact meas-
urement. Scientometrics, 91(2), 461–471.
Lin, J., & Fenner, M. (2013a). The many faces of article-level metrics. Bulletin of the American Society for
Information Science and Technology, 39(4), 27–30.
Lin, J., & Fenner, M. (2013b). Altmetrics in evolution: Defining and redefining the ontology of article-level
metrics. Information Standards Quarterly, 25(2), 20–26.
Leydesdorff, L., Wagner, C. S., & Bornmann, L. (2019a). Interdisciplinarity as diversity in citation patterns
among journals: Rao-Stirling diversity, relative variety, and the Gini coefficient. Journal of Informet-
rics, 13(1), 255–269.
Leydesdorff, L., Wagner, C. S., & Bornmann, L.(2019b).Annex to:Interdisciplinarity as Diversity in Cita-
tion Patterns among Journals: Rao-Stirling Diversity, Relative Variety, and the Gini coefficient,
from.https://​leyde​sdorff.​github.​io/​diver​sity_​measu​rement/
Mcallister, P. R., Narin, F., & Corrigan, J. G. (1983). Programmatic evaluation and comparison based on
standardized citation scores. IEEE Transactions on Engineering Management, 30(4), 205–211.
Merton, R. K. (1973). The sociology of science. Theoretical and empirical investigations. University of Chi-
cago Press.
Melero, R. (2015). Altmetrics a complement to conventional metrics. Biochemia Medica, 25, 152–160.
https://​doi.​org/​10.​11613/​BM.​2015.​016
Michaela, M., Miriam, O., & Šušol, J. (2021). Bibliometrics versus altmetrics: Researchers’ attitudes in Slo-
vakia. Iberoamerican Journal of Science Measurement and Communication. https://​doi.​org/​10.​47909/​
ijsmc.​11
Miura, T., Asatani, K., & Sakata, I. (2021). Large-scale analysis of delayed recognition using sleeping
beauty and the prince. Applied Network Science. https://​doi.​org/​10.​1007/​S41109-​021-​00389-0
Mohammadi, E., Thelwall, M., Kwasny, M., & Holmes, K. L. (2018). Academic information on Twitter: A
user survey. PLoS ONE, 13(5), 0197265. https://​doi.​org/​10.​1371/​journ​al.​pone.​01972​65
Mohammadi, E., Gregory, K. B., & Thelwall, M. (2020). Which health and biomedical topics generate the
most Facebook interest and the strongest citation relationships? Information Processing and Manage-
ment, 57(3), 102230.
NISO. (2016). NISO RP-25–2016 Outputs of the NISO Alternative Assessment Project. https://​groups.​
niso.​org/​highe​rlogic/​ws/​public/​downl​oad/​17091
O’Connor, E. M., Nason, G. J., et al. (2017). Newsworthiness vs scientific impact: are the most highly
cited urology papers the most widely disseminated in the media? BJU International, 120(3), 441–
454. https://​doi.​org/​10.​1111/​bju.​13881
Parker, J., & Van Teijlingen, E. (2012). The Research Excellence Framework(REF): Assessing the
impact of social work research on society. Practice, 24(1), 41–52.
Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Altmetrics: A manifesto. Retrieved March 20,
2020 from the http://​altme​trics.​org/​manif​esto/
Priem, J., Piwowar, H., & Hemminger, B. (2012). Altmetrics in the wild: Using social media to explore
scholarly impact. Retrieved March 20, 2020 from the https://​arxiv.​org/​abs/​1203.​4745v1/

13
Scientometrics

Qin, F., & Gao, J. (2019). Comparative analysis of Altmetrics and citation measurement based on the
Scopus database. Journal of the China Society for Scientific and Technical Information, 38(4),
377–383.
Qiu J., & Yu H. (2015). Stratifying Altmetrics Indicators Based on Impact Generation Model. Interna-
tional Conference on Scientometrics and Informetrics.
Ravenscroft, J., Liakata, M., Clare, A., & Duma, D. (2017). Measuring scientific impact beyond academia:
An assessment of existing impact metrics and proposed improvements. PLoS ONE, 12(3), e0173152.
Redner, S. (2005). Citation statistics for more than a century of physical review. Physics Today, 58(6),
49–54.
Rinia, E., Van Leeuwen, T., Bruins, E., et  al. (2001). Citation delay in interdisciplinary knowledge
exchange. Scientometrics, 51(1), 293–309. https://​doi.​org/​10.​1023/A:​10105​89300​829
Roemer, R. C., & Borchardt, R. (2015). Meaningful metrics: A 21st century librarian’s guide to biblio-
metrics, altmetrics, and research impact. Association of College and Research Libraries, A division
of the American Library Association.
Stern, N. (2016). Lord Stern’ s review of the Research Excellence Framework: call for evidence. https://​
excha​nge.​notti​ngham.​ac.​uk/​conte​nt/​uploa​ds/​Stern-​review.​pdf
Stirling, A. (2007). A general framework for analysing diversity in science, technology and society.
Journal of the Royal Society Interface, 4(15), 707–719.
Sun, J., Min, C., & Li, J. (2016). A vector for measuring obsolescence of scientific articles. Scientomet-
rics, 107(2), 745–757.
Szczurko, O., Cooley, K., Busse, J. W., Seely, D., Bernhardt, B., et  al. (2007). Naturopathic care for
chronic low back pain: A randomized trial. PLoS ONE, 2(9), 919. https://​doi.​org/​10.​1371/​journ​al.​
pone.​00009​19
Taberner, R. (2018). Altmetrics: Beyond the impact factor. Actas Dermosifiliogr (english Edition), 109,
95–97. https://​doi.​org/​10.​1016/j.​ad.​2018.​01.​002
Teixeira, A. A. C., Vieira, P. C., & Abreu, A. P. (2017). Sleeping Beauties and their princes in innova-
tion studies. Scientometrics, 110(2), 541–580.
Thelwall, M., Haustein, S., et al. (2013). Do altmetrics work? Twitter and ten other social web services.
PLoS ONE, 8(5), e64841.
Thelwall, M., Kousha, K., Dinsmore, A., & Dolby, K. (2015). Alternative metric indicators for fund-
ing scheme evaluations. Aslib Journal of Information Management, 68(1), 2–18. https://​doi.​org/​10.​
1108/​AJIM-​09-​2015-​0146
Thelwall, M. (2018). Early Mendeley readers correlate with later citation counts. Scientometrics, 115(3),
1231–1240. https://​doi.​org/​10.​1007/​s11192-​018-​2715-9
Thelwall, M. (2021). Alternative medicines worth researching? Citation analyses of acupuncture, chiro-
practic, homeopathy, and osteopathy 1996–2017. Scientometrics, 126(1), 8731–8747. https://​doi.​
org/​10.​1007/​s11192-​021-​04145-0
van Raan, A. F. J. (2004). Sleeping beauties in science. Scientometrics, 59(3), 467–472.
van Raan, A. F. J. (2015). Dormitory of physical and engineering sciences: Sleeping beauties may be
sleeping innovations. PLoS ONE, 10(10), e0139786.
van Raan, A. F. J. (2017). Sleeping beauties cited in patents: Is there also a dormitory of inventions?
Scientometrics, 110(3), 1123–1156.
van Raan, A. F. J., & Winnink, J. J. (2018). Do younger Sleeping Beauties prefer a technological prince?
Scientometrics, 114(2), 701–717.
van Raan, A. F. J. (2021). Sleeping beauties gain impact in overdrive mode. Scientometrics, 126(6), 1–22.
Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics,
10(2), 365–391.
Wang, X., Mao, W., Xu, S., & Zhang, C. (2014). Usage history of scientific literature: Nature metrics and
metrics of Nature publications. Scientometrics, 98, 1923–1933.
Wang, X. G., Lv, T., & Donald, H. (2019). How do altmetric sources evaluate scientific collaboration? An
empirical investigation for Chinese collaboration publications. Library, 38(3), 563–576.
Wu, S. N., & Zhao, R. Y. (2016). Research on status and development trend of altmetrics tools. Document,
Information & Knowledge, 33(1), 84–93. https://​doi.​org/​10.​13366/j.​dik.​2016.​01.​084
Zahedi, Z., Costas, R., & Wouters, P. (2014). How well developed are altmetrics? A cross-disciplinary anal-
ysis of the presence of “alternative metrics” in scientific publications. Scientometrics, 101, 1491–1513.
Zingg, C., Nanumyan, V., & Schweitzer, F. (2020). Citations driven by so-cial connections? A multi-layer
representation of coauthorship networks. Quantitative Science Studies, 1, 1493–1509. https://​doi.​org/​
10.​1162/​qss_a_​00092

13
 Scientometrics

Zhai, S. S., Ye, D. L., Hu, P., et al. (2020). Evaluation of the academic impact of data papers fused with
altmetrics and citation analysis. Journal of the China Society for Scientific and Technical Information,
39(7), 710–718.
Zhang, H. H., & Ye, F. Y. (2020). Identifying “associated-sleeping-beauties” in “swan-groups” based on
small qualified datasets of physics and economics. Scientometrics, 122(3), 1525–1537.
Zhang, L., Rousseau, R., & Glänzel, W. (2016). Diversity of references as an indicator for interdisciplinarity
of journals: Taking similarity between subject fields into account. Journal of the Association for Infor-
mation Science and Technology, 67(5), 1257–1265.
Zhang, L., & Wang, J. (2018). Why highly cited articles are not highly tweeted? A biology case Scientomet-
rics, 117(1), 495–509.
Zhang, L., Sun, B. B., & Wang, X. W. (2020). The impact of interdisciplinarity: distinct effects on usage and
citation. Journal of the China Society for Scientific and Technical Information, 39(5), 469–477.
Zhao, W., Korobskiy, D., & Chacko, G. (2020). Delayed recognition: the co-citation perspective. Frontiers
in Research Metrics and Analytics, 5, 577131. https://​doi.​org/​10.​3389/​frma.​2020.​577131
Zong, Z., Liu, X., & Fang, H. (2018). Sleeping beauties with no prince based on the co-citation criterion.
Scientometrics, 117(3479), 1841–1852. https://​doi.​org/​10.​1007/​s1119​2018-​2932-2

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under
a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted
manuscript version of this article is solely governed by the terms of such publishing agreement and applicable
law.

13