Using bibliometrics in research evaluation: An Introduction

Ann Kushmerick Manager, Research Evaluation and Bibliometric Data

Agenda
Introduce you to the background of bibliometrics Show examples of how bibliometrics are used in research evaluation Explain a variety of bibliometric indicators and what they are used to measure Allow you to ask questions.

Thomson Reuters (formerly ISI) has been the authority on citation data for over 50 years.
Science Citation Index Social Sciences Citation Index Arts & Humanities Citation Index WOK 4.0 ISI Web of Knowledge

1960 2007

1965 2008

1970

1975

1980

1985

1990

1995

2000

2005

SciSearch PC-based citation sets Custom citation projects and national indicators mainframe Essential Science Indicators PC-based Indicators for journals, nations, institutions Century of Science TS Innovation

Trends in scholarly research

Competition for government research funding increasing Available funding decreasing Competition for top research faculty is on the rise Accountability:
Research spending Demonstrating return on investment (ROI)

Proving the institutions quality of research to:

Prospective students Prospective faculty members/research staff Investors/donors

Result: Institutions seek objective data on research performance, for data-based decision making

Why do you need to evaluate your R&D?

Know where youve been, where you are today, and where you want to be in the future. Use your precious resources more effectively
Strategic planning Researcher-level assessments- tenure review, etc. Reward those who are outstanding, and identify & support those who are falling behind Report progress to government bodies, boards of directors/ trustees, funding agencies, etc. Plan a research strategy for each program Recruit the best staff

Why use bibliometrics?

Bibliometrics: the statistical analysis of bibliographic data, mainly in scientific and technical literatures. It measures the amount of scientific activity in a subject category, journal, country, topic or other area of interest.
Dr. Henry Small, Chief Scientist, Thomson Reuters (Scientific)

If you can measure that of which you speak, and can express it by a number, you know something of your subject; but if you cannot measure it, your knowledge is meager and unsatisfactory - Lord Kelvin (William Thomson)

Why citation metrics?

The primary and most common way to quantitatively track and measure research outcomes. Uses data on peer reviewed journals and citations received by those articles. Citations are an indicator of an articles impact and usefulness to the worldwide research community; they are the mode by which peers acknowledge each others research. Citation metrics are: Transparent Repeatable
7

Citation metrics are one piece of the research performance puzzle. They complement other types of assessment.

Citation data Funding data

Awards/Honors Peer review

Combination of multiple methodologies and data sources Examples of data sources / methodologies:
# of and value of Grants awarded # of awards (e.g. Nobel Prizes) Peer evaluation Publication counts Citation counts/citation metrics

Combination of factors
None of these measure works perfectly on its own, there are always anomalies and human judgment is required to interpret the results
8

Validation Studies: Citation Frequency and Its Correlation with Other Measures of Peer Esteem
Name Kenneth E. Clark Jonathan R. Cole & Stephen Cole Henry G. Small Julie A. Virgo Michael E.D. Koenig Eugene Garfield Charles Oppenheim Andy T. Smith & Michael Eysenck Field Psychology Physics Collagen research Cancer research Pharmaceutical research Nobel Prize winners University rankings (RAE) Psychology Year 1957 1967, 1973 1977 1977 1983 1992 Mid 1990s2002

Typical findings, r = .7 to .9
Smith and Eysenck, comparing 1996 and 2001 RAE scores given to psychologists at 38 UK universities (peer review) with their citation counts, concluded: The two approaches measure broadly the same thing.

Different stakeholders in research evaluation

External Entities
Government agencies/funding organizations /accreditors/ state government

Institutional Management
Management, including committees, provost, vice provosts

Institutional Depts/Divisions
Deans; Department Heads Institutional assessment, academic affairs

Library

Individuals
Faculty, staff

Citation metrics are only as good as their source

Web of Science data have been used in major research evaluation initiatives around the globe for decades.

US National Science Foundation

Science & Engineering Indicators

European Commission: European Union

Science & Technology Indicators

US National Research Council -- Doctoral Program Ranking Also used by government entities in France, Australia, Italy, Japan, UK, Portugal, Norway, Spain, Belgium, South Korea, Canada, and more, to shape higher education policy.

Web of Science- the first and largest citation index

Selectivity and control of content- high, consistent standards Depth- 100+ years- including cited references Consistency and reliability- ideal for research evaluation e.g. field averages Unmatched expertise- 40+ years of citation analysis and research evaluation Multidisciplinary- Science, Social Science, Arts and Humanities 11,000+ journals and 716 million+ cited references Conference Proceedings- 12,000 conferences
12

Source and Foundation:

Data
Thomson Reuters Expertise and Processing Address Unification Research Analytics Resources

Web of Science

Data Cleansing & Standardization Normalization and Baselines

For many years Thomson Reuters has provided a wide range of tools and
services supporting accurate and effective research evaluation.

Our specialists work with Web of Science data and ensure maximum standardization and unification before delivery to customers.

Thomson Reuters offers not just simple counts and averages, but real metrics founded on baselines for comparison and normalized statistics.

The H-index

H-index: This statistic reflects the number of papers (N) in a given dataset having N or more citations. 26 of M.A. Marras 88 papers were cited at least 26 times each.

H-index

15

Information about using the h-index

In some cases, the h-index better reflects overall performance than an average citation rate, which can be highly skewed by one very highly cited paper. H-index is dependent on field of research and age of researcher- only compare like with like E.g. a young researcher is at a disadvantage Bounded by the total number of works an author has published If youve only published 15 works, no matter how highly cited they are, your h-index will never exceed 15. Some say it doesnt adequately take into account total citation count received, or give enough weight to highly cited papers (see example in the following slide)
16

These three authors have the same h index (4) but very different publication histories and total citation counts.
Paper
Dr. Smith 1 2 3 4 5 6 7 8 9 10 h-index Total citations 4 273 4 92 80 80 61 43 4 4 1

Citation Count
Dr. Jones 80 4 4 4 Dr. Zhang 10 9 8 4 4 1 1 1 1 1 4 40

There are alternatives to the h-index that help to mitigate these issues. We can calculate those for you on a custom basis.

JOURNAL IMPACT FACTOR- metric that reflects a journals overall performance

2005 2006 2007 2008 2009

All Previous Years

2008 Impact Factor

Citations Source paper published in 2008 Cited reference published in 2006 or 2007

Ratio of citations from 2008 (to papers published in 2006 and 2007) to papers published in 2006-2007
Nature Genetics

Improper use in evaluation: Judging papers, people

Journal IFs were not designed for or intended to be used as a measure or proxy of performance for individual papers or authors Skewed nature of citation distribution means that in most cases IF over-estimates impact of papers, people publishing in a journal Field differences affect IF. Comparisons across fields are flawed. Summing up or combining IFs exacerbates this.

Types of citation metrics and what they measure

Productivity # papers Relative Impact/ Bench-marking Journal actual/expected citation rate Category actual/ expected citation rate Percentile in category and mean percentile % papers in top 10% of their field % papers in top 1% of their field Aggregated Performance Indicator Specialization Disciplinarity index Interdisciplinarity index
20

Total influence # citations H-index Efficiency Avg. citation rate

Percent of papers cited

Can be applied to an institution, a researcher, a research group, etc.

What is a researchers impact?

Steven Boonen Department of Experimental Medicine KUL

21

http://www.kuleuven.be/cv/u0009433e.htm

Dr. Boonens most cited paper. What is a good citation count?

Androgens and bone, Endocrine Reviews, 2004

2nd generation citations: indirect, long term impact. Citations received by the papers that cited this paper.
22

Dr. Boonens most cited paper. What is a good citation count?

Androgens and bone, Endocrine Reviews, 2004

Average citations specific to journal, year of article, and document type. Baseline for journal performance. Boonens ratio of actual cites to expected: 0.88 (112/127.73)
23

Dr. Boonens most cited paper. What is a good citation count?

Androgens and bone, Endocrine Reviews, 2004

Average citations specific to category, year of article, and document type. Baseline for category performance. Boonens ratio of actual cites to expected: 3.8 (112/29.47)

24

Dr. Boonens most cited paper. What is a good citation count?

Androgens and bone, Endocrine Reviews, 2004

Percentile position specific to field and year. The closer to zero, the more highly cited. Boonens paper falls into the top 0.85% of all endocrinology/metabolism papers from 2004.

25

Two fields, two different citation distributions

Mean 10

Chemistry in 2003- more citation activity than physics

<50%

>50%

>10%

>1%

Mean 7.5

Physics in 2003

<50%

>50%

>10%

>1%
26

Number of citations

Dr. Boonens most cited paper. What is a good citation count?

Androgens and bone, Endocrine Reviews, 2004

Annual breakdown of citations received

27

Dr. Boonen- What is a good citation count?

Productivity # papers Total influence # citations Self-citing papers H-index Efficiency 149 1,136 57 22 Relative Impact/ Benchmarking Journal actual/expected Category actual/expected Mean percentile % papers in top 10% of their field 1.14 1.48 33.91 19%

Avg. citation 7.62 rate Percent of 43 papers cited

% papers in top 1% 5% of their field

Special- ratios compare this Journal and category performance Disciplinarity index 0.97 ization researchers citation counts to the norms or expected rates in his discipline. Boonen is performing above average on both the journal and category levels (1.14 ,1.48).

Mean percentile reflects the percentile performance of the researchers work, on average. e.g. in top 33.91 percent

28

% of papers above or below the expected number at various percentile thresholds

1.0 represents the average

4.26% of Boonens papers exceed the expected level at the 1% threshold

29

All metrics and analyses can be performed on any subset of the data
An entire institution One researcher Group of researchers (e.g. a department) Field of research within your institution Topic of research within your institution A collaboration partnership within your institution

30

Normalized measures are key to understanding performance: compare researchers, groups, etc.
Total Total Papers Cites Dr. S Boonen 149 102 22 601 2,193 1,136 1,492 74 6,478 39,624 Avg. Cites H per Paper Index 7.62 14.63 3.36 10.78 18.07 22 17 5 38 Journal actual/ expected Category Mean actual/ expected Percentile

(endocrinology, KUL)

1.14 1.89 1.14 1.05 -

1.48 2.20 1.08 1.01 1.18

33.91 53.89 40.74 44.11 -

Dr. JM Kaufman
(endocrinology, Ghent Univ)

Dr P De Grauwe Dr ..P De Grauwe

(economics, KUL)

KUL- all endocrinology Belgiumendocrinology

These metrics normalize for field, age, and size 31 differences.

Comparative data
Rank and compare countries and institutions Drill down to a specific research discipline

32

Analyze countries: Netherlands Specialization: top 6 fields

-2% +7%

-3%

33

Analyze National trends in paper output/production

Netherlands: 35% increase over 10 years

34

On a relative scale, 1 represents the world average citation rate in clinical medicine. All of these countries perform above both the world and EU baselines.

35

Analyze Institutions:

Benchmark against peer institutions or aspirational peers US government institutions in Physics

Impact relative to field average- a normalized metric All of these organizations are currently performing above both the world and US averages.

36

How does my institution compare to peer institutions or aspirational peers?

Article count

World share of articles in field

37

Compare article output of institutions from various countries

Univ British Columbia has the highest output currently

38

Compare impact for institutions from various countries

Penn State Univ has the highest relative impact currently

All of these universities are currently performing above the world average in plant/animal science, although some have increased and some have decreased over time.

39

Compare performance across fields using relative impact

On a relative scale, we can see that Kings College London performs above the world average in all of these fields, which have varying average citation rates.

40

Trend analysis: Do your institutions policy changes correlate with a measureable result?
Changes in staffing/funding might correlate with changes in publication/citation counts

Why did citations drop between 1993 and 1996?

Analyze outcomes of collaborations Facilitate effective collaboration, domestic and global

Which institutions do we collaboration frequently with in genetics?

Which collaborations have the most impact in this field (category performance ratio)?

Publication and citation analysis: The 10 commandments

1. Consider whether available data can address question the 1. e.g. the large 2. Choose field definitions, publication 2. datasetyearsthedata basic types, and and of more 3. Decide on whole or fractional counting the research, theuses 3.PublicationReuters more Thomson types: usually

reliable the articles and reviews 4. Judge whether data require editing to remove artifacts data whole counting. analysis. 4.Arifacts- misleading -Considerthe field of 5. Compare like with like The GoldenFields:name and the Rule understand 5. Consider

journal categories address counts 8.just absolutethe fieldfollow a power Citation patterns of paper; research; variants 6. Use relative measures, not 6. Compare to age and Years: usually -consider negative atpapers law distribution. Manyofleast 5 and the career length and self get 7. Obtain multiple measures journal averages. Use years citations cited few or no the researcher. times, and few normalized measures to 8. Recognize skewed nature of citation data papers disparate entities, compare are cited many times. 9. Confirm data collected areand 10. to question 9 relevant Like all good science, review results 10. Ask whether the results are e.g. physics researcher to reasonable medical researcher and ask further questions.

Thank you
Bibliometrics white papers
http://isiwebofknowledge.com/media/pdf/UsingBibmetrics-WP.pdf http://isiwebofknowledge.com/media/pdf/UsingBibliometricsinEval_WP.pdf http://isiwebofknowledge.com/incites/ http://isiwebofknowledge.com/products_tools/analytical/