| 0

Evidence Based Evaluation

Alexander van Servellen
Consultant, Research Management (SEA)
Elsevier Singapore

UNIMAP, Malaysia
February 24th, 2016

| 1


1. Introduction to SciVal and Evidence Based Evaluation

2. Analysis and Reporting

3. Non-Performance Variables

4. Setting up SciVal

5. Accessing SciVal

6. Q&A

| 2

Introduction SciVal and
Evidence Based Evaluation

| 3

Typical uses
Measure scientific
production and
benchmark research
performance at
multiple levels

Assess the
Monitor research
international impact
of research

Map collaboration
Identify leading
networks and
organizations and
identify collaboration

Assess the impact of
research funding on Identify who is doing
the scientific output of what and with whom
researchers and in a variety of fields
graduate students

Source: http://www.science-metrix.com/en/expertise/bibliometrics/know-how

| 4

Main purposes of metrics

Evaluation of performance
• Typically “top-down” performed by an authority
• Essential to account for variables that can affect metrics values besides
differences in performance, so that the evaluation is fair

Demonstration of excellence
• Typically “bottom-up” performed by entity requesting resources
• Generally aiming to find a way to showcase strengths compared to peers;
non-performance variables may be used to advantage in this case

Scenario modeling
• Non-performance variables may or may not be important depending on
• e.g. if modelling recruitment in nanoscience, might not need to worry about
different citation rates between fields because only looking at one field!

and specific research topics. and evaluate them. 5 Data updated weekly from Scopus (same data source as the University Rankings use and used in many studies about research performance) . | 5 SciVal is an online tool for evaluating research. You can easily create teams. departments. SciVal offers quick. easy access to the research performance of 220 nations and 6000 research institutions worldwide.

| 6 Reporting at various levels Top Research Office Management Faculty admins Research Unit Admins Researchers .

23 million patents from 5 patent offices worldwide All content is vigorously vetted by an independent. 15-person. | 7 The data in SciVal comes from Scopus Scopus is the largest abstract and citation database of peer-reviewed research literature from around the world. government researchers and corporate R&D professionals who need a comprehensive and efficient place to search. Its the core data source of Elsevier Research Intelligence solutions. and used by academics. discover and analyze research. international board of experts called the Content Selection and Advisory Board (CSAB) More than 3. Over 21.900 titles from more than 5.000 customers worldwide in all geographic regions .000 international publishers and 105 different countries Over 54 million records.

| 8 Scopus content 10% 8% 5% 5% 2% 1% Other 60% 1% 1% 1% 1% 1% 1% 1% 1% .

| 9 .

| 10 Through SciVal and Scopus. Elsevier has experience with metrics .

| 11 Elsevier’s metrics approach in a nutshell Figure 1 of http://www.com/online-tools/research-intelligence/resource-library/resources/response-to-hefces-call-for-evidence-independent-review-of-the-role- of-metrics-in-research-assessment .elsevier.

| 12 Levels of aggregation .

| 13 Everything .

| 14 Region .

| 15 Country .

| 16 University .

Department or Research Centre . | 17 Faculty.

| 18 Individual .

| 19 Notes .

| 20 Analysis & Reporting .

one analysis) . It 3 or 5 year period. group of metrics. have created. it allows you to looks at specifically at co- style reports based on a select a time period publication trends. You can top collaboration partners institutions. countries in filter. You anywhere from 1996 to allows you to identify the can analyze countries. and compare country. applied here as a subject overview of one entity at institutions. | 21 Now that you have customized SciVal – you can explore and analyze using SciVal Overview module Benchmarking module is Collaboration module provides dashboard flexible.e. It gives a can compare researchers Research areas can be comprehensive with departments. one month ago. as well as researchers and across different types of identify potential research areas you entities (i. you directly collaboration partners. a time. any choose from over 17 of an institution or researcher.

snowballmetrics. m) Number of Citing Countries Collaboration (geographical) Citation Impact metrics Academic-Corporate Collaboration Citation Count Citations per Publication Cited Publications Usage metrics h-indices (h. g. m) Views Field-Weighted Citation Impact Views per publication Publications in Top Percentiles Field-Weighed Views Impact Publications in Top Journal Percentiles Disciplinarity metrics Collaboration Impact (geographical) Journal count Academic-Corporate Collaboration Impact Journal category count Snowball Metric.com/metrics . www. | 22 20 sets of metrics at your disposal Slice and dice your data from multiple angles to identify your core strengths and weaknesses Productivity metrics Collaboration metrics Scholarly Output Authorship Count h-indices (h. g.

| 23 Notes .

| 24 Non-Performance Variables .

Instead the difference can be caused by non-performance variables. . | 25 When making comparisons. we must take into consideration “Non-performance variables” Size Discipline Publication-type Database coverage Manipulation Time In some cases the difference in a metric value between two entities (authors for example) might not reflect a difference in performance.

| 26 Publications types receive differing levels of citations Because some publication types are cited more often than others. we should not compare different types without applying normalization. .

we should not compare papers from different fields without applying normalization. | 27 Citation practices differ between disciplines Because some subject areas are cited more often than others. .

| 28 Size and Time Citation count is directly related to volume Do you think the declining trend is indicative of a decrease in quality or is it caused by a non- performance variable? CPP normalizes for differences in publication volume FWCI normalizes for subject area. and time . document type.

| 29 Notes .

| 30 Setting Up SciVal .

) . | 31 My SciVal – Researchers and Groups • Creating Researchers • Creating Groups of researchers (departments etc.

| 32 Master List .

| 33 First create a Master List of researcher names .

| 34 We start with the authors for who you do not know the profile. or have more than one profile in Scopus .

| 35 Go to My SciVal and “Define a new Researcher” .

| 36 Do this only for the authors who have more than one Scopus Author ID!! add the Scopus author ID field twice If you do not know the profile number. then search by name Fill in the Author IDs of one author .

or validate the publications first . You can save the researcher. | 37 SciVal will ask you to confirm merging of profiles. in cases where one researcher has more than one profile.

so that once there are hundreds of researchers in SciVal. each one is clearly tagged. . | 38 Add a tag to the researcher indicating which department he/she belongs to Here I added a tag.

| 39 Notes .

| 40 Bulk Import Researchers that have one Scopus Author Profile .

Past just the ID. . | 41 The researchers with one Scopus Author ID can be bulk imported into SciVal You can paste the author IDs belonging to one department in the box. not the author name etc.

then you should search for this author manually by name in SciVal. | 42 SciVal will show you if any Author ID is not found If an author ID is not found. .

Remember to tag the researchers. Give the group a name Check this box to create a group for these researchers Tag the researchers . | 43 Create the Department and give it a name.

Use the sharing function to give others access to this department . | 44 Now you have created a department in SciVal which you can easily share with other SciVal users.

| 45 Groups can be combined into larger organizational units Drag and drop units to combine .

| 46 Notes .

| 47 My SciVal – Research Areas • Creating Researcher Areas using keywords • Creating Research Areas by grouping entities .

| 48 Create Research Area .

| 49 Use Keywords to define a very specific area of science Useful search tips .

| 50 Refine & Validate Validate that publications are representative of topic .

| 51 Prepare your Research Areas in advance. Areas larger than 5000 publications can take 48 hours .

| 52 Accessing SciVal .

If you do not have a username and password for SciVal. then please register here.com Who does not have access to SciVal? Please raise your hand.scival. . | 53 Accessing SciVal: go to www.

com/research-intelligence .vanservellen@elsevier.elsevier. | 54 Thank you very much Alexander van Servellen a.com www.