You are on page 1of 17

Scientometrics (2010) 82:333–349

DOI 10.1007/s11192-009-0043-9

Use of Scopus and Google Scholar to measure social


sciences production in four major Spanish universities

Goio Etxebarria Æ Mikel Gomez-Uranga

Received: 25 June 2008 / Published online: 10 June 2009


 Akadémiai Kiadó, Budapest, Hungary 2009

Abstract A large part of Social Sciences and the Humanities do not adapt to international
proceedings used in English for scientific output on databases such as the Web of Science
and Scopus. The aim of this paper is to show the different results obtained in scientific
work by comparing Social Sciences researchers with those of other sciences in four
Spanish universities. The first finding is that some Social Sciences researchers are some-
what internationalised. However, the majority of individuals who are prestigious in their
local academic-scientific community do not even appear on the information sources
mentioned above.

Keywords Social Sciences production  Visibility on international databases 


Spanish universities

Introduction

The standards used to measure scientific work are evolving over time. The Institute of
Scientific Information (ISI) citation databases, accessible on the WEB OF SCIENCE
(WoS) have been used for the last 40 years (Jacso 2005; Bosman et al. 2006; Meho and
Yang 2007). SCOPUS (Reed Elsevier 2004) and GOOGLE SCHOLAR (GS) (Payne 2004)
were introduced in November 2004. Only the latter can be used free of charge. From its
characteristics, as well as the role of Google in the Internet, we deduce that it has a very
good chance of becoming the most adequate standard for the Internet era.
The information on Scopus databases exceeds that of WoS since they store 27 million
abstracts and quotes and feature 14,500 journals (in comparison to 8,700 on WoS).
Elsevier, the owner of Scopus, writes up abstracts which are provided to the its main users
while the Web of Science only uses the abstracts furnished for each article (Jacso 2005,
p. 1539). The contents of these multidisciplinary databases are increasing. For instance,
WoS, still the leader in classical areas such as Physics and Chemistry, is including

G. Etxebarria (&)  M. Gomez-Uranga


University of the Basque Country, Bilbao, Spain
e-mail: goio.etxebarria@ehu.es

123
334 G. Etxebarria et al.

documents on conference proceedings in its database. On the other hand, Scopus, the
database we will refer to in this paper, is more efficient for fields like Health which are
extremely important at present (Jacso 2005).
A great deal has been written about the citation system as an evaluation method (Meho
and Yang 2007, p. 2; Bauer and Bakkalbasi 2005). Our aim in this article is to analyse the
validity of the previously mentioned databases for application in certain disciplines and in
places where academic work is not carried out in English.
The WoS is still the most widely used and well known source in the academic world. It
is also the oldest, and to some extent, the least developed, so it receives a great deal of
criticism. This source is almost exclusively limited to titles in English, thus overestimating
the importance of work in some branches of science while underestimating output in others
such as Social Sciences and the Humanities. Nor has it included citations from books or
most conference proceedings until now. Furthermore, it is more careless with the use of
synonyms, names, initials, etc. than other databases (Meho and Yang 2007). Books and
monographs are especially important in many fields of the Social Sciences, which leads us
to think that those disciplines are improperly valued by WoS. Authors such as Cronin et al.
1997 demonstrate, for instance, that there are two types of Sociology researchers: those
who publish papers and those who publish monographs.
It is not totally relevant to compare the Scopus list of journals with the WoS list because
new material and possibilities are constantly being included. According to data furnished
on the webpage, Scopus accesses over 2,800 titles in Social Sciences. Google Scholar is
quite different from WoS and Scopus because it covers a different realm of information
although it does coincide in some fields. The literature revised showed that GS was
inefficient in the following aspects: (a) Extracting and organising information on GS is
tedious. (b) The list of documents is excessive, for instance, lists of courses and references.
(c) Citations from the same conferences sometimes appear in different ways while they are
more standardised and organised on WoS and Scopus (Meho and Yang 2007, pp. 2110–
2111).
These drawbacks make systematic searches with complete efficient results difficult on
GS. One must detect mistakes and contrast, to the greatest possible degree, with the other
two information sources. However, in this article we propose this tool as the best to view a
broader reality when dealing with different sciences and more universal as far as local
academic communities are concerned.

Objective

The objective of this article is to measure scientific output in Social Sciences. Numerous
studies and personal empirical findings have led us to state that the evaluation criteria for
researchers’ output in these disciplines have given rise to a great deal of debate. This is due
to the difficulty in reaching consensus on these points within the scientific community,
which transfers its knowledge in extremely different ways.
A study by Jacso (2005) showed that Social Sciences journals (including Economics
and Psychology) on the Scopus database accounted for 17% at the end of 2005. However,
Social Sciences documents accounted for only 2% of the total. This gap cannot be
explained only by documents from these disciplines being included in other journals
related to different scientific disciplines.
The results of scientific output are not properly reflected on the most widely used
databases (WoS, Scopus) at the national and regional levels (Gómez et al. 2006). This

123
Use of Scopus and Google Scholar to measure social sciences production 335

situation is more severe in Social Sciences and the Humanities where a variety of docu-
ments which are not articles (books, monographs, academic documents, theses, research
projects, etc.) are produced and not included in these information sources (De Moya 2007).
The most evident case can be seen in Law, where most of the output is monographs.
Scopus is a more adequate database to include documents that are not found in journals,
which would prompt a better way of dealing with Social Sciences. However, Scopus was
still weaker in Sociology than WoS in some exceptional cases in 2006 (Bosman et al.
2006).
Social Sciences and the Humanities do not strictly adapt to the standardised interna-
tional proceedings for scientific output (Harzing 2008). In Spanish universities most of the
results from these disciplines are transferred to papers which are not published in inter-
nationally homologated journals in English. This makes it difficult to measure research
productivity through the amount of citations. Literature on the subject points out that
journals in English are over represented (Macroberts and Macroberts 1996) and that the
disciplines of Social Sciences and the Humanities are not sufficiently included (Moed
2005). The quality standards for Social Sciences and the Humanities are, in part, more
complex. WoS and Scopus are still far from capturing the diversity which exists in the field
of Social Sciences in the world. There is a dark area on the scientific planet, which makes it
practically impossible to access output published at the local level; which Gibbs (1995)
calls ‘‘lost science in the third world’’.
The aim of this study is to analyse the difference in results for scientific output in the
field of Social Sciences in non-English speaking areas in comparison to other sciences
(mostly Experimental and Health Sciences). We also aim to put forth some proposals that
may help to evaluate results of work in Social Sciences.

Method

We have selected a sample group of researchers from four Spanish universities for this
study. The following universities were selected: Universidad Complutense (Complutense
University) in Madrid (UC), Universidad de Barcelona (University of Barcelona) (UB),
Universidad Autónoma de Barcelona (Autonomous University of Barcelona) (UAB) and
the Universidad del Paı́s Vasco (University of the Basque Country) (UPV/EHU). The first
three are among the leading Spanish universities in international and European rankings
(QS WORLD UNIVERSITY RANKINGS 2008; Webometrics Ranking of World Uni-
versities 2008; Academic Ranking of World Universities 2008). While the University of
the Basque Country is not included in the ranking of top Spanish universities like the other
three (UC, UB, UAB), it is in an intermediate to high position. Its size (number of staff and
students) and the structure of major fields taught are similar to the others. All the branches
and disciplines are taught as in the other three. Furthermore, the authors of this paper have
detailed knowledge of the Basque University. These reasons and the compatible com-
parison prompted us to keep it in our study.
We used the report titled ‘‘La universidad española en cifras’’ (‘‘Spanish Universities in
Figures’’) (Hernández 2006) to know the number of academic staff and researchers at the
universities studied in this article. They were classified by teaching fields: the Humanities;
Social Sciences and Law; Experimental Sciences; Health Sciences and Technical Sciences.
We calculated the percentage of Social Sciences and Humanities researchers in comparison
to the total number of researchers at each university, except for the University of the
Basque Country since these data are not included in the report. This information enables us

123
336 G. Etxebarria et al.

to contrast total scientific output and Social Sciences output at each university with the
total number of teaching staff and the number of Social Sciences staff (including
Humanities staff). Thus, we can properly value the differences that may appear for the
amount of scientific output included in the Scopus database.
Three samples were taken; two for Social Sciences researchers and another for
researchers who are most visible on international databases. As we shall see, the latter
group was used to contrast the results of the first sample groups.
When drawing up the samples we have included the following disciplines in Social
Sciences: Law, History, Geography, Sociology, Political Science, Communication Sci-
ences and Economics (Economics, Econometrics and Finance; Business, Management and
Accounting). Our samples do not include Psychology, Philosophy, Anthropology, Litera-
ture, Linguistics, …
When putting together the samples of Social Sciences authors, we decided to limit it to
certain disciplines. Psychology covers a large number of fields, sometimes in contact with
Social Sciences, and many times related to Health Sciences and Clinical Medicine.
Therefore, we are not arguing about the Social Science nature of this discipline. However,
when searching on Scopus and following the established selection criteria, we realised that
a great majority of the journals and documents appearing had high medical content and
were characterised similar to Health Sciences. On the other hand, the aim of this paper is
not to analyse certain disciplines, but to focus on the most significant group in Social
Sciences.
The sample groups of Social Sciences researchers were obtained in two ways:
– On the one hand, a group of Social Sciences researchers was chosen from each
university. This was based on their recognition by the local-national scientific
community in their disciplines.
– On the other hand, a sample group of authors was chosen from each university. Choice
was based on the documents found on Scopus, making use of double criteria: authors
with the highest number of documents and authors whose most cited document appears
amongst the leading ones in Social Sciences. These double criteria were used to ensure
selection of researchers with the highest level of exposure in their field, and therefore,
the most relevant.
We used the following procedure to obtain the sample of prestigious researchers who
are not visible on the international scene:
1. A survey answered by experts in the fields of Social Sciences we selected.
The survey was completed by 12 professors from the four universities studied. All of
them had many years of experience in the university (over 25 years) from all the fields of
knowledge pertaining to the Social Sciences we selected in this article.
Based on their experience and knowledge of their academic community, they were
asked to: (a) choose the four people they consider to be the most prestigious and referential
in their scientific community; (b) choose 4 professors from each of the four universities
and, if possible, put them in order from most to least prestigious.
As far as the results of the survey are concerned: A relevant part of the researchers
mentioned in the survey responses have been included in the final sample shown in this
article. Some of the researchers suggested in the responses have not been included in the
final sample either because they did not meet our selection requirements (especially when
studying them on Google Scholar) or because more researchers were chosen than those
making up the sample.

123
Use of Scopus and Google Scholar to measure social sciences production 337

2. Participation in scientific boards and institutions, publication in Spanish journals, and


evaluation of researchers by evaluation agencies.
The third sample group, taken from researchers who are most visible on international
databases, was obtained from the Scopus database, applying the same double criteria
mentioned above.
Information from the researchers was obtained mainly from the Scopus database and
through Google Scholar. This search was carried out in May 2008. Google Scholar and
Scopus data took about 500 h of collecting and processing time.

Scopus database

In the first stage of this study, we mainly made use of the Scopus database. This was
because it includes a larger number of journals and documents than WoS while also
offering ‘‘Affiliation Search’’. This enabled us to focus our scope of study on the output of
researchers in the four universities we had selected.
All centres, faculties and departments, which sometimes show different affiliation
identification numbers (AF-ID), were included for each university. In this manner, we
obtained the total number of documents that Scopus identifies with authors from each
university.
Different searches were done on Scopus to yield the double sample:
– A sample of referential researchers for the Sciences as a whole. All the documents from
each university included in the database were put in order by the number of times they
were cited (‘‘Cited by’’). We drew up a small sample of authors from each university
based on the double criteria: authors with the highest number of documents and authors
whose most cited document appears amongst the leading ones.
– Sample of Social Sciences researchers with output included in this database. A search
of documents (‘‘Advanced Search’’) in the field of Social Sciences was carried out for
each university. The Scopus Subject Areas which we include in the sample group for
Social Sciences researchers are: Social Sciences (Scopus features a Social Sciences
subject area within the general category of Social Sciences), Economics (Economics,
Econometrics and Finance), Business (Business, Management and Accounting),
Decision Sciences, and Arts and the Humanities. We used the same double selection
criteria as in the previous sample: authors with the highest number of documents or
authors whose documents are most cited in Social Sciences. In this case, we did not
take into account documents and authors that are not included in field we have chosen
for this study (for instance, documents from Psychology or Anthropology). However,
we have kept documents from fields like Mathematics because the contents that appear
in this search are related to Economics or other Social Sciences.
Once the double sample was obtained, each of the researchers was analysed by an
author search in order to know details of each individual profile and, in particular, the
number of documents by each author on the Scopus database and which document was
most cited. There were 26 authors for Experimental and Health Sciences (Table 5).
Authors with over 100 documents or 500 citations in the most cited document were chosen.
The sample of Social Sciences authors who are significant on the Scopus database included
40 (Table 4); having selected authors who had over five documents or who had over ten
citations in the most cited document.

123
338 G. Etxebarria et al.

We also carried out an author search to locate researchers who are prestigious in their
scientific community in Social Sciences in order to know which documents from their
scientific output are included in this database.

Google Scholar

We used GS to find out the actual scientific output of prestigious researchers in the field of
Social Sciences as their work is barely mentioned on international databases. We collected
the following information for each author: documents (books, chapters in books, papers at
conferences, academic reports, etc.) with more than one citation, articles with more than
one citation, the number of citations in the most cited document, total number of citations
(taking into account documents with more than two citations) and documents in English.
It was not our intention for the different sample groups to show a ranking of the authors
in which all of the authors who are mentioned have a higher number or documents and/or
citations than those who are not included in the sample. We are aware that numerous
authors who are not in our sample accomplish the criteria were set. However, the sample
group had to be limited in order to make this study feasible. Sample groups with more
authors and universities could have been drawn up although we believe that the results
would not have varied sufficiently to change the conclusions and proposals we have put
forth.

Results

When comparing the number of documents from the fields of Social Sciences and the
Humanities found on the Scopus database with the total number of documents for each
university included in the study, the results were quite similar. We found that results
ranged from 5.12% for the University of Barcelona to 6.78% for the University of the
Basque Country, yielding an average of 5.78% for the four universities included in the
study (Table 1). Initially, scientific output in Social Sciences seems quite low when
compared to total scientific output.
These results must be compared with the relative importance of Social Sciences
researchers in the different universities in order to interpret them correctly. Table 2 shows
the percentage of teaching and research staff classified by the five branches of teaching in
relation to the total number of staff at each university.

Table 1 No. of documents found on Scopus for each university


University Total number No. of social % of social sciences
of documents sciences documents documents/total
no. of documents

Universidad Complutense (UC) 27,369 1,512 5.52


Universidad de Barcelona (UB) 31,245 1,600 5.12
Universidad Autónoma de Barcelona (UAB) 23,257 1,500 6.45
UPV/EHU 12,115 822 6.78
Total (UC ? UB ? UAB ? UPV) 93,986 5,434 5.78
Source: Scopus

123
Use of Scopus and Google Scholar to measure social sciences production 339

Table 2 Percentage of teaching and research staff classified by state universities and teaching fields (2004)
University Humanities Social sciences Experimental Health Technical Total
and law sciences sciences sciences

UC 15.23 33.16 22.51 19.49 9.62 100


UB 17.05 34.89 16.85 28.36 2.86 100
UAB 16.61 30.32 16.61 29.99 6.46 100
AVERAGE (UC, 16.15 33.10 19.28 24.84 6.63 100
UB, UAB)
Source: Data furnished from the report by Hernández (2006). This report includes information from 2004; it
does not offer information on the University of the Basque Country (UPV/EHU)

Taking these data into account, Social Sciences and Humanities teaching and research
staff represent, on the average, nearly half of the total staff at the four universities in our
study (49.25%). However, the percentage of documents published in these areas and
included on Scopus averages only 5.78% of the total number of documents from these
universities included on the database. In other words, half of the researchers do not produce
even 6% of the scientific output shown on this database. The figure is significant and
clearly demonstrates the difference between the amount of Social Sciences output on
international databases from these universities in comparison to output in other fields of
science, mostly Experimental and Health Sciences, as we shall see.
In this article, we also analyse the gap shown on the international level for Social
Sciences researchers at Spanish universities. We drew up a sample group of 30 researchers
from the four universities in the study, based on their position and prestige in their different
local-national academic communities.
The results obtained with GS are shown in Table 3. It clearly indicates the significant
number of documents (books, monographs, conferences, etc.) (with more than one citation)
these authors have produced. In some of Social Sciences disciplines such as History or
Law, scientific and academic output is transmitted in these types of documents. However, it
is very uncommon in other fields of science such as Experimental and Health Sciences.
In some cases, the total number of citations is considerable although the citations
included in GS are very different from those found on the WoS and Scopus databases. The
latter make special reference to peer-reviewed journals.
The table also shows that there are few documents in English and many authors do not
have a single one. This is an enormous contrast with the productive capacity shown in the
local-national language. Table 3 includes the amount of documents by these authors that
can be found on Scopus. We observe that very few authors in the sample group have a
document on the Scopus database.
In the case of this group of researchers, the h Index calculation, an indicator which is
used increasingly to measure scientific output in some disciplines, would vary greatly
according to the source used for the calculation: WoS, Scopus or GS. However, we think
that the index would not be significant in this case. It is important to note, nevertheless, that
such a high output like that of most of the authors in the study requires a long active career.
It is also important to bear in mind that researchers in the field of Social Sciences may
obtain considerable results with their scientific and academic work even when they are
quite old.
During the search for each author in the sample group, we noticed that quite a large
amount of the work in the books section is signed by individuals. This led us to the

123
340 G. Etxebarria et al.

Table 3 Sample of Social Sciences authors acknowledged as prestigious members of the local scientific
community in their discipline
Author University Area A B C D E F G

Abellan, Joaquin L. UC Political Sciences 6 1 94 126 – –


Bustamante, Enrique UC Communication Sciences 14 6 91 383 3 2 2
Garcı́a De Enterria, UC Law 23 6 45 76 – –
Eduardo
Rojo, Luis A. UC–BANCO Economics 13 13 18 130 1 –
ESPAÑA
Segura, Julio UC Economics 13 12 55 281 – –
Anes, Gonzalo UC Economic History 18 2 55 85 – –
Artola, Miguel UC History 31 8 43 421 – –
Pérez Moreda, UC History 9 4 34 81 2 –
Vicente
Alvarez Junco, José UC History 15 1 12 64 3 –
Fusi, Juan Pablo UC History 18 2 37 230 1 –
Alberti Rovira, Enoch UB Constitutional Law And Political 6 1 8 25 – –
Sciences
Fontana, Josep UB History 41 7 61 525 1 –
Nadal, Jordi UB History 34 11 130 603 2 –
Carreras, Albert UB History 20 8 93 464 3 –
Tugores, Joan UB Economics 5 2 74 155 – –
Botella, Joan UAB Political Sciences 13 4 21 136 2 –
Subirats, Joan UAB Political Sciences 36 12 162 636 3 5 4
Vallés, Josep M. UAB Political Sciences 12 1 52 176 4 1 2
Miguelez, Faustino UAB Sociology 8 3 84 189 – –
Moragas, Miquel UAB Journalism and Communication 7 1 17 59 1 –
Sciences
Etxezarreta, Miren UAB Economics 12 7 26 160 – 1 1
Jauregui, Gurutz UPV/EHU Law 9 2 25 102 2 –
Bazo, Ma Teresa UPV/EHU Sociology 10 11 48 191 1 2 4
Pérez Agote, Alfonso UPV/EHU–UC Sociology 13 5 45 143 1 –
Zunzunegui, Santos UPV/EHU Communication Sciences 10 3 126 203 – –
Garitaonandia, UPV/EHU Journalism and Communication 8 5 22 97 3 1 8
Carmelo Sciences
Ibarra, Pedro UPV/EHU Political Sciences 16 2 58 148 1 1 0
De La Granja, José UPV/EHU History 12 0 16 69 – –
Luis
González Portilla, UPV/EHU History 15 1 11 43 – –
Manuel
Fernández Pinedo, UPV/EHU History 16 6 12 69 – –
Emiliano
A Books, chapters of books, papers from conferences, reprotes, etc. with more than one citation on Google
Scholar, B Articles with more than one citation on Google Scholar, C No. of citations for the most cited
document on Google Scholar, D Total no. of citations (taking into account documents with more than two
citations) on Google Scholar, E References in English found on Google Scholar, F No. of documents on
Scopus, G No. of citations for the most cited document on Scopus
Source: Information from Scopus and Google Scholar

123
Use of Scopus and Google Scholar to measure social sciences production 341

conclusion that a great deal of the output in Social Sciences is individual rather than team
work, especially for books and monographs.
There are Social Sciences researchers whose work is more highly internationalised than
others. For our study, we drew up a sample of 40 authors with this profile in the four
universities, following the criteria mentioned in the section titled ‘‘Method’’. In Table 4,
we observe that Economics, with its specialised branches: Applied Economics, Econo-
metrics, Economic Policy, Treasury, Business Economics, Labour Economics … stands
out from the other disciplines. Even authors that work in other areas such as Mathematics
or Operational Research use quantitative methods in their work for economic analysis. The
fact that Environmental Sciences and Economics have been closely related in the last
decades is obvious. Therefore, practically 35 researchers (almost 90%) of the sample
would be listed in the field of Economics. In other words, the field of Social Sciences with
the highest exposure on the international level is Economics.
It is important to note that not all branches of Economics have the same level of
exposure. As we saw in Table 3, there are well known economists who work in subfields
which have little or no international exposure, like a large part of the Spanish scientific
community from the field. In the sample, Geography has a reasonable percentage (10%)
(Table 4) of researchers with a more international profile. The modern concept of Geog-
raphy is clearly influenced by an Anglo Saxon vision of the subject.
However, the most meaningful part of the sample is what we cannot see; in other words,
the disciplines that do not have a single representative: Law, History, Political Science,
Communication Sciences, etc. These fields are very important at the local-national level
and, as we saw in Table 3, have reached a high level of academic-scientific output.
However, they do not appear on international databases which are used to measure sci-
entific output in other fields of science.
We used the criteria explained in ‘‘Method’’ to draw up a sample group of 26
researchers from the four universities (Table 5) to contrast some of the results we had
obtained previously. In other words, this was based on their scientific output shown on the
Scopus database. Experimental Sciences (Physics, Chemistry, Cell Biology…) and Health
Sciences (Medicine, Pharmacology) are the leading fields in output while neither Social
Sciences, nor the Humanities or Technical Sciences can be seen. Output from the former
fields is totally internationalised with global and highly interrelated scientific communities
that transfer their knowledge exclusively through articles (in English) signed by a large
number of authors. This clearly indicates the existence of international work groups (and
often means that the researchers are highly mobile).
We will specifically compare this sample with that obtained for Social Sciences authors
with the greatest amount of work shown on the database (Table 4); in other words, with
authors having the most international profile. The difference between the amount of output
(number of documents found on Scopus) and the number of citations from the most cited
document is very large. Information on Social Sciences researchers is much less visible.
However, we also observe this difference within the field of Social Sciences, as we pointed
out earlier and this is even true for the different disciplines in Experimental and Health
Sciences. These comparisons are only valid for similar or related fields of science and are
not reliable for very different fields.
Thus, we can state that scientific output from different fields of knowledge must be
measured very carefully. The different guidelines followed to transmit knowledge must be
taken into account as well as the unequal impact that different disciplines and subdisci-
plines have, depending on their size and the critical mass of the scientific community being
examined.

123
342 G. Etxebarria et al.

Table 4 Sample of Social Sciences authors with visibility on the Scopus database
Author University Area Doc. on Citations
Scopus Scopus

Artalejo, Jesus R. UC Mathematics 56 46


Sosvilla-Rivero, Simón UC Economics 35 25
Montero, Javier UC Mathematics 32 14
Domı́nguez Rodrigo, Manuel UC Prehistory 27 34
Gutiérrez Puebla, Javier UC Human Geography 11 49
Pérez-Amaral, Teodosio UC Economics 10 14
Herce, Jose A. UC Economics 8 10
Costa-Font, Joan UB Economics 41 15
Suriñach, Jordi UB Economics 26 80
Martin-Vide, Javier UB Geography 23 60
Satorra, Albert UB–POMPEU Economics 18 121
FABRA
Artis, Manuel UB Economics 18 18
Rafels, Carles UB Economics 18 4
Perelló, Josep UB Physics–Finance– 17 16
Maths
Carrion-I-Silvestre, Josep UB Economics 17 15
Lluis
Solé-Ollé, Albert UB–UAB Economics 10 16
Roca, Jordi UB–UAB Economics 7 14
Calonge, Samuel UB Economics 5 150
Garcia-Ramon, Ma Dolors UAB Geography 30 18
Perez-Castrillo, David UAB Economics 29 26
Massó, Jordi UAB Economics 24 18
Martinez-Alier, Joan UAB–ICTA Environmental 23 71
Sciences
Munda, Giuseppe UAB–ICTA Environmental 23 63
Sciences
Calvó-Armengol, Antoni UAB Economics 21 36
Barberá, Salvador UAB Economics 20 33
Macho-Stadler, Inés UAB Economics 19 25
Grifell-Tatjé, Emili UAB Economics 13 46
Caballé, Jordi UAB Economics 13 17
Esteban, Joan UAB Economics 11 46
Brandts, Jordi UAB–CSIC MADRID Economics 10 21
Rialp, Alex UAB Economics 7 55
De La Fuente, Angel UAB Economics 7 47
Padilla, Emilio UAB–UB Economics 7 16
Priestley, Gerda K. UAB Geography 4 23
Zarzuelo, Jose Manuel UPV/EHU Economics 19 22
Valenciano, Federico UPV/EHU Economics 19 18
Barcena-Ruiz, Juan Carlos UPV/EHU Economics 15 9
Espinosa, Ma Paz UPV/EHU Economics 10 15
Gomez Uranga, Mikel UPV/EHU Economics 6 147

123
Use of Scopus and Google Scholar to measure social sciences production 343

Table 4 continued

Author University Area Doc. on Citations


Scopus Scopus

Gardeazabal, Javier UPV/EHU Economics 6 40

ICTA Institute of Environmental Science and Technology, CSIC Centro Superior de Investigaciones
Cientı́ficas, Doc. on Scopus No. of documents on Scopus, Citations Scopus No. of citations for the most
cited document on Scopus
Source: Information from Scopus

Discussion and proposals

The results explained lead us to question just how universal and uniform the evaluation of
scientific output is in all branches of knowledge. When applying the same standards and
criteria used for Experimental Sciences to Social Sciences, we have seen that the results are
extremely unclear for the actual output and dedication to research of staff in those disci-
plines. Obviously, the researcher profile varies in the different fields and they follow very
different parameters to transmit their scientific output (books, monographs, papers, reports,
conferences, etc.), not limiting themselves to publication of articles in international jour-
nals which are usually in English.
In the results we have observed that there are at least two researcher profiles in the broad
field of Social Sciences. Authors whose work is visible on international data bases (mainly
in the field of Economics) seem to have profiles very similar to Experimental Sciences
researchers. However, the amount of documents published and number of citations are quite
different. Other Social Sciences researchers, on the other hand, are barely mentioned on
international databases although they are recognised in their local-national scientific
community for their considerable prestigious work, normally in the local-national language.
Our observations are coherent with a study on scientific activity carried out by subjects
between 1999 and 2003 measured on WoS. This international database showed that Spain
participated with the following amount of documents: 3.1% for Physics; 4.19% for
Chemistry; 4.43% for Microbiology; 2.64% for Economics and Business; 0.84% for Social
Sciences and 0.08% for Law. When measured in terms of citations, all the fields showed
lower percentages in comparison to the number of documents except Social Sciences.
Results were Physics with 1.24% Chemistry with 1.01%; Microbiology with 0.83%,
Economics and Business with 0.67%; Social Sciences with 0.87%; and Law with 0%
(Gómez et al. 2006, p. 285). We believe that such a marked difference between citations
and documents may be partially because Spanish researchers are on the periphery in
relation to the critical mass of research now being carried out mainly in the U.S. and the
U.K. This situation was also mentioned for Latin America and the Caribbean in a study by
Collazo-Reyes et al. (2008).
The matter of citations is even more marked when examining local journals. As Col-
lazo-Reyes et al. (2008, p. 158) point out, ‘‘More than 50% of the papers published by
Latin American and Caribbean journals were found to be uncited in comparison with the
33% reported for the same types of papers published by Latin American scientists in the
mainstream literature from 1973 to 2005. This implies that papers sent by local, regional
and external researchers to Latin American and Caribbean journals are less likely to be
cited than those sent to other mainstream titles.’’
The most referential databases (WoS, Scopus) do not take the cultural context and
values which scientific output is based on into consideration (Osca-Lluch et al. 2008;

123
344 G. Etxebarria et al.

Table 5 Sample group of authors from the fields of Experimental Sciences and Health Sciences
Author University Area Doc. on Citations
Scopus Scopus

Hernando, Antonio UC Physics 458 187


Vallet-Regı́, Marı́a UC Inorganic Chemistry 291 205
Martı́n, Nazario UC–UAM Organic Chemistry 280 489
Martı́nez-Frı́as, Ma Luisa UC–Instituto Salud Carlos III Genetics 251 97
Zapata, Agustı́n G. UC Cell Biology 159 554
Lizasoain, Ignacio UC Pharmacology 117 805
Rodés, Joan UB Medicine 715 734
Morante, Joan Ramon UB Electronics 493 112
Solans, Xavier UB Inorganic Chemistry 479 124
Gomis, Ramón UB Medicine 264 1376
Ruiz-Lapuente, Pilar UB Astronomy 57 3037
Rozas, Julio UB Genetics 30 1262
Grauges, Eugeni UB–UAB Physics 315 282
Garrido, Lluis UB–UAB Physics 305 282
Baselga, Jose Ma UAB–Hospital Vall D’hebron Medicine 194 2329
Pedro-Botet, Juan Carlos UAB Medicine 147 161
Nogués, Josep UAB Physics 125 1222
Echenique, Pedro Miguel UPV/EHU–DIPC Physics 277 230
Barandiaran, Jose Manuel UPV/EHU Physics 273 65
Rojo, Teófilo UPV/EHU Inorganic Chemistry 226 314
Goñi, Félix M. UPV/EHU Biochemistry 220 332
Asúa, Jose Ma UPV/EHU–POLYMAT Chemical Engineering 195 159
Rubio, Ángel DIPC (UPV/EHU) Physics 178 446
Palacios, Francisco UPV/EHU Organic Chemistry 105 61
Sanchez-Portal, Daniel DIPC (UPV/EHU) Physics 69 1125
Zubiaga, Ana Ma UPV/EHU Genetics 29 503
DIPC Donostia International Physics Center, POLYMAT Instituto Universitario de Materiales Poliméricos
(UPV/EHU), UAM Universidad Autónoma de Madrid (Autonomous University of Madrid), Doc. on Scopus
No. of documents on Scopus, Citations Scopus No. of citations for the most cited document on Scopus
Source: Information from Scopus

Garfield 1979). ‘‘In 2004, the WoS only included 29 Spanish Medical and Experimental
Sciences journals, 2 Social Sciences journals and 15 Humanities journals (the latter of
which were only partially included)’’ (Gómez et al. 2006, p. 277). A complementary
indicator can be used in Spain for Social Sciences and the Humanities: the Spanish Index
for Social Sciences and the Humanities (ISOC) (Gómez et al. 2006, p. 277). In the case of
Latin America and the Caribbean, only eight journals were included in the Science Citation
Index (SCI) in 1985. While it had improved for 1995–1996, few local journals were seen:
12 journals in the SCI and 16 in the Social Sciences, Arts and Humanities Citation Indexes
(Collazo-Reyes et al. 2008, p. 146).
Other complementary indexes must be proposed to set reasonable limits on the
unstoppable pace of Anglo Saxon science. These must have a broad linguistic and cultural
scope, for instance, Chinese and Spanish. In this sense, the Chinese Social Sciences
Citation Index and the Chinese Science Citation Database are being used (Xin-Ning et al.

123
Use of Scopus and Google Scholar to measure social sciences production 345

2001; Leydesdorff and Bihui 2005), as well as the Iberian Evaluation Index for Scientific
Publications (Buela-Casal et al. 2003). These indicators are used to complement the ISI
databases (WoS). Some examples include SciELO (Scientific Electronic Library Online)
developed in Brazil (Meneghini and Packer 2007), and Redalyc in Mexico (an on line
scientific periodicals and newspaper library for Latin America, the Caribbean, Spain and
Portugal) (Aguado-López and Rogel-Salazar 2006).
The number of citations from papers depends greatly on the scope of the journal in
which they are published Collazo-Reyes et al. (2008, p. 146). Thus, citations from articles
published in local non-English journals will mainly come from the local scientific com-
munity. The number of citations will be determined by the size of the community. Lists of
quotes are related, in part, to what scientific communities receiving authors belong to. If
the scientific-academic community is local, the citations will tend to be from local
researchers. However, if the communities authors belong to are more international and
furthermore, authors are at leading Anglo Saxon centres, then the list of citations will most
likely come from all parts of the international community.
When drawing up a local standard, language must be carefully handled to ensure it can be
assumed by international communities. In spite of the fact that a great deal of citations are
from local sources (both from the academic community perspective as well as territory), we
believe that the endogamy suggested by Collazo-Reyes et al. (2008) should be commented
on since English may be the main language used by authors who cite the documents of other
authors, as well as the language of the document cited. Undoubtedly, some of the citations
used in local communities will be in the local language (although there will be more quotes in
English). The article by Collazo-Reyes et al. (2008, p. 156) clearly demonstrates the situation
we are describing since citations from articles in English and directed to articles in English
are more numerous than those in Spanish (as the second language for local literature).
The search for a new standard follows a rapidly changing path. There is an increasing
number of documents in English in comparison to Spanish on databases (Gómez et al.
2006, p. 278). In some cases, journals is Spanish are discontinued and the local scientific
community has a higher level of exposure. Some policies to aid creation of high quality
local journals and financial backing may improve the visibility and international impact of
local communities (Bonilla and Pérez-Angon 1999).
Echeverrı́a et al. (2006) put forth some points to improve the situation of academic
research communities in the Humanities which may coincide with some of the ideas
explained in this article. One of the points proposed by the authors included: ‘‘Backing for
journals published in Spain that comply with international quality standards; as well as
journals and publications that may reach this standard, based on a previously drawn up
project to achieve this (…). Carry out a study on quality indicators for the Humanities that
can become a standard for different evaluation processes (…). Define quality criteria for
Humanities journals and publications’’ (Echeverrı́a et al. 2006, p. 323).
In previous sections of this article, we have highlighted the almost complete dominance
of academic journals in English when evaluating scientific output. In this sense, we can
state that there are many documents which are invisible in the conventional evaluations of
the Institute for Scientific Information (ISI) (Kousha and Thelwall 2008). Google Scholar
broadens the scope for measuring scientific output in two directions in comparison to other
databases: (a) the diversity of documents that it takes into account; and (b) the linguistic
diversity it envisages.
GS gathers information on documents that are not only published in journals but others
such as: papers from conferences, books, theses, research reports and preprint repositories
(Google Scholar 2008). The documents offered on GS which do not come from journals may

123
346 G. Etxebarria et al.

become quite relevant. For instance, technical reports make up the most common output in
engineering and conference papers top the list in Computer Sciences (Harzing 2008).
Social Sciences, except for Psychology and part of Economics, are not widely included
on conventional ISIs because many of the documents from these fields are not published in
journals. For example, books and monographs are the preferred formats for legal experts
and historians. At present, we can state that the disciplines that benefit most from the
information gathered by GS are: Engineering, Computer Sciences, Economics and Busi-
ness Administration and Management, Mathematics, Social Sciences, Art and the
Humanities (Harzing 2008).
GS has the potential to offer a wider panorama of world scientific output in other
languages. However, the different languages are more influential in some disciplines than
in others. In their study of sample articles from four disciplines, Kousha and Thelwall
(2008, p. 284) show that 36% of the citations in Chemistry are in Chinese as well as 23% of
the total articles while 96% of the articles in Physics are in English.
It is in the field of Social Sciences and the Humanities that a large part of academic
work is presented in local languages (non-English). In this article, we have demonstrated
that a great deal of the output by Social Sciences researchers in four Spanish universities is
not written in English. We believe that this statement can be extended to the entire Spanish
scientific community for these disciplines, with exceptions.
A proposal to move towards standards for local scientific-academic communities in the
field of Social Sciences could take the following considerations as reference points:
1. Phases of transition focused on achieving greater international exposure could be
envisaged, implementing different time frameworks for the various scientific fields and
subfields. Our study sheds light on researchers with the highest level of exposure in
Social Sciences. By fields, aid and incentives could be set up to improve researchers’
international exposure. This transition would not necessarily lead to discontinuing
non-English output, even in the long term.
2. Parallel to this process, ‘‘homologated journals’’ should be increased in nearly every
field of Social Sciences and the Humanities, responding to the challenge of meeting
hypothetical international quality control. In other words, this would involve making
use of peer-review, citation systems, etc. with the addition of new features. For
instance, inclusion of researchers who do similar work for prestigious international
journals on the editorial board and among evaluation staff for the local language
journal, even if the cost of translating articles had to be added. Outside exposure could
also be improved by including abstracts in English. It would also be advisable that
local journals not be demanded to comply with the international citations requirement
so that they could be included on the ISI or Scopus databases.
3. Governments (jointly with local academic groups) would have to have the capacity to
negotiate conditions with Elsevier, Thomson or Google to envisage a larger amount of
non-English documents and citations.
4. Consensus should be reached between different local scientific-academic groups to
provide their governments with ad hoc indicators that would allow for combinations of
international and ‘‘homologated’’ local output, re-adapting the indicators over time.
5. It must be assumed that the amount of citations, and therefore, the impact indexes of
local output without exposure and the possible homologated output in accordance with
the criteria we explained in previous parts of this article will never reach the same
production levels as in English. It is similar to what happens on local markets in
comparison to more internationalised ones.

123
Use of Scopus and Google Scholar to measure social sciences production 347

Conclusion

In this article, we have focused on showing that the profile for Social Sciences researchers
is different from that of their counterparts in other sciences (mainly in comparison to
Experimental Sciences). This calls for implementation of complementary mechanisms to
evaluate their scientific output. This is especially relevant in the case that the output is
transmitted in a language other than English, which is the case for the majority of the
Social Sciences community in Spain.
Transition towards greater international exposure for Social Sciences in Spanish uni-
versities should be a method to achieve better quality output in these disciplines. One way
to progress in this direction would be increasing the number of ‘‘local homologated
journals’’ complying with international quality standards. In any case, greater international
exposure would have to be supported by adequate public policy.

Acknowledgements Well-chosen comments on an earlier draft from two anonymous referees are grate-
fully appreciated. We are most grateful to Carmen Acebal, Research Vice-Rector of the Universidad
Complutense, and to Manu Barandiaran, Iñaki Lasagabaster, Jon Barrutia, Pedro Ibarra, Juan Carlos Miguel,
Alex Arizkun and Jon Arluzea, professors and researchers, for helpful information on the academic and
research practices in different scientific areas and disciplines. The authors would particularly wish to thank
Carmen Pérez (CSIC) and Nekane Zaldua (UPV/EHU) for their assistance with data collection. Lastly, we
would like to give special thanks to our translator, Patricia O’Connor, for her excellent work translating,
revising and correcting this article and previous ones.

Appendix

See Tables 6, 7.

Table 6 No. of teaching and research staff classified by universities. 2004


University Total

UC 5,896
UB 4,517
UAB 3,034
UPV/EHU 4,541

Source: Data were obtained from a report directed by Hernández (2006), except in the case of the University
of the Basque Country (UPV/EHU), which is not included in the report. Information on this university
comes from its webpage and dates from 2006

Table 7 No. of teaching and research staff classified by state universities and teaching fields (2004)
University Humanities Social sciences Experimental Health Technical Totsal
and law sciences sciences sciences

UC 898 1,955 1,327 1,149 567 5,896


UB 770 1,576 761 1,281 129 4,517
UAB 504 920 504 910 196 3,034
Total (UC ? UB ? UAB) 2,172 4,451 2,592 3,340 892 13,447
Source: Information furnished from the report directed by Hernández (2006). No data offered for the
University of the Basque Country (UPV/EHU)

123
348 G. Etxebarria et al.

References

Academic Ranking of World Universities. (2008). Shanghai Jiao Tong University. Retrieved May 20, 2008,
from http://www.arwu.org.
Aguado-López, E., & Rogel-Salazar, R. (2006). Redalyc: red de revistas cientı´ficas de Ame´rica Latina, el
Caribe, España y Portugal: un balance a tres años de camino. Buenos Aires, Argentina: Consejo
Latinoamericano en Ciencias Sociales: Fraga Jorge CLACSO, pp. 209–233. Available in http://www.
clacso.org.ar/biblioteca.
Bauer, K., & Bakkalbasi, N. (2005). An examination of citation counts in a new scholarly communication
environment. D-Lib Magazine, 11(9). Retrieved November 14, 2007, from http://www.dlib.org/dlib/
september05/bauer/09bauer.html .
Bonilla, M., & Pérez-Angon, M. A. (1999). Revistas Mexicanas de Investigación Cientı́fica y Tecnológica.
Interciencia, 24, 102–106.
Bosman, J., Van Mourik, I., Rasch, M., Sieverts, E., & Verhoeff, H. (2006, June). Scopus reviewed and
compared. Utrecht University Library.
Buela-Casal, G., Fernández-Hermida, J. R., & López, W. (2003). Iberindex: Índice Iberoamericano de
evaluación de publicaciones cientı́ficas. Infocop, 84, 41–44.
Collazo-Reyes, F., Luna-Morales, M. E., Russell, J. M., & Pérez-Angón, M. A. (2008). Publication and
citation patterns of Latin American & Caribbean journals in the SCI and SSCI from 1995 to 2004.
Scientometrics, 75(1), 145–161.
Cronin, B., Snyder, H., & Atkins, H. (1997). Comparative citation rankings of authors in monographic and
journal literature: A study of sociology. Journal of Documentation, 53(3), 263–273.
De Moya, F. (2007). Indicadores Bibliométricos de la Actividad Cientı´fica Española (1990–2004). FECYT.
Retrieved November 29, 2007, from http://sise.fecyt.es.
Echeverrı́a, J., Maltrás, B., & Sequera, R. (2006). La investigación en humanidades: Problemas especı́ficos.
In J. Sebastián & E. Muñoz (Eds.), Radiografı´a de la investigación pública en España (pp. 303–325).
Madrid: Biblioteca Nueva.
Garfield, E. (1979). Citation indexing. Its theory and application in science, technology and humanities.
New York: Wiley.
Gibbs, W. W. (1995). Lost science in the Third World. Scientific American, 273, 76–83.
Gómez, I., Sancho, R., Bordons, M., & Fernández, M. T. (2006). La I ? D en España a través de publi-
caciones y patentes. In J. Sebastián & E. Muñoz (Eds.), Radiografı´a de la investigación pública en
España (pp. 275–302). Madrid: Biblioteca Nueva.
Google Scholar. (2008). Retrieved May 16, 2008, from http://scholar.google.com/scholar/about.html.
Harzing, A. W. (2008). Google Scholar—a new data source for citation analysis. Retrieved April 2, 2008,
from http://www.harzing.com/pop_gs.htm.
Hernández, J. (2006). La Universidad española en cifras, Informe 2006. Available in http://www.crue.org/
pdf/CRUE%202006.
IN-RECS, Índice de impacto, Revistas Españolas de Ciencias Sociales. http://ec3.ugr.es/in-recs.
ISI. http://www.isiwebofknowledge.com.
Jacso, P. (2005). As we search—comparison of major features of the Web of Science, Scopus, and Google
Scholar citation-based and citation-enhanced databases. Current Science, 89(9), 1537–1547.
Kousha, K., & Thelwall, M. (2008). Sources of Google Scholar citations outside the Science Citation Index:
A comparison between four science disciplines. Scientometrics, 74(2), 273–294.
Leydesdorff, L., & Bihui, J. (2005). Mapping the Chinese Science Citation Database in terms of aggregated
journal–journal citation relations. Journal of the American Society for Information Science and
Technology, 56, 1469–1479.
Macroberts, M. H., & Macroberts, B. R. (1996). Problems of citation analysis. Scientometrics, 36(3),
435–444.
Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: Web
of Science versus Scopus and Google Scholar. Journal of the American Society for Information Science
and Technology, 58(13), 2105–2125.
Meneghini, R., & Packer, A. (2007). Is there science beyond English? EMBO Reports, 8, 112–116.
Moed, H. F. (2005). Citation analysis in research evaluation. New York: Springer.
Osca-Lluch, J., Blesa, P., Barrueco, J. M., Velasco, E., & Krichel, T. (2008). Some aspects of citation
indexes in Spain: A comparative analysis. Scientometrics, 75(2), 313–318.
Payne, D. (2004). Google Scholar welcomed. News from The Scientist, 5(1):20041123-01. Retrieved
October 31, 2006, from http://www.the-scientist.com/news/20041123/01/.
QS World University Rankings. (2008). QS quacquarelli symonds. Retrieved May 16, 2008, from http://
www.topuniversities.com.

123
Use of Scopus and Google Scholar to measure social sciences production 349

Reed Elsevier. (2004). Scopus comes of age. Retrieved October 31, 2006, from http://www.reed-
elsevier.com/index.cfm?Articleid=1075.
SCOPUS. http://www.info.scopus.com.
UPV/EHU. http://www.ehu.es.
Webometrics Ranking of World Universities. (2008). Consejo Superior de Investigaciones Cientı́ficas
(CSIC). Retrieved May 16, 2008, from http://www.webometrics.info.
Xin-Ning, S., Xin-Ming, H., & Xin-Ning, H. (2001). Developing the Chinese social science citation index.
Online Information Review, 25, 365–369.

123