Professional Documents
Culture Documents
Responsible
Research
Assessment
Global Research Council (GRC)
Conference Report 2021
A virtual conference from the
Global Research Council | held in November 2020
1
Responsible Research Assessment
2
GRCGRC
Conference
Conference
Report
Report
2021
2021
CONTENTS
Executive Summary............................. 4
Context.................................................. 6
Key analytics from the conference..... 8
Thematic analysis...............................12
1 Research assessment shapes research culture.......................................... 13
3
Responsible Research
Responsible Research Assessment
Assessment
“funders can
“global approaches
kick-start other
“global collaboration must be mindful of
changes in the
is essential to RRA” local context, culture
assessment system”
and language”
4
GRC Conference Report 2021
1
Research assessment influences how research is performed and disseminated
Research What funders value and measure will influence what is valued in the research
assessment shapes ecosystem
research culture Funders can initiate positive culture change through careful design and
implementation of research assessment
As stewards of the R&I system, funders must address barriers to cultural change,
especially pressures to perform in university league tables.
2
Diverse R&I sectors A funder’s definition of research excellence needs to be multidimensional
Research excellence should encompass perspectives and experiences from
create high-quality people of all backgrounds
research and impact Funders must support diversity through a broad portfolio of funding.
3
Fostering a healthy Funders should employ clear criteria, relevant indicators, and regular
self-evaluation
R&I system that Funders should experiment with new processes and test that they have the
does not undermine desired outcomes
Addressing bias and providing equal opportunities are essential to the
diversity integrity of the research assessment process and should not be hindered by a
resistance to change.
4
Approaches must be mindful of local context, culture, language and
Research and unintended consequences which impact other countries
scholarship are Funders across the globe should agree on what conditions are required to
transnational support a healthy, vibrant R&I system
Funders should work together to achieve high level coordination on what is
endeavours valued and assessed as collaboration can deliver systemic change.
5
All stakeholders in All stakeholders should collaborate to develop and evaluate RRA and resist
shifting blame to other parties
the R&I ecosystem Resistance to change should be countered through rewarding responsible
play a vital role in assessments
Buy-in for new approaches can be achieved through co-creation.
its construction
1. https://rori.figshare.com/articles/report/The_changing_role_of_funders_in_responsible_research_assessment_progress_
obstacles_and_the_way_ahead/13227914 5
Responsible Research Assessment
Context
UK Research and Innovation (UKRI) in The GRC is a virtual network of research funding agencies
dedicated to the promotion of sharing data and best
collaboration with the UK Forum for practices for high-quality collaboration among funding
Responsible Research Metrics and the agencies worldwide. The virtual conference replaced a
face-to-face meeting due to take place as a satellite
National Research Foundation (NRF) conference to the GRC’s 9th Annual Meeting in Durban in
in South Africa, delivered a GRC Virtual May 2020, which was postponed due to COVID-19.
Conference on RRA during 23-27 The conference was aimed primarily at senior
representatives from GRC participating organisations
November 2020. and therefore focused on RRA as undertaken by funders
(while considering their role within a wider ecosystem).
The conference was an opportunity to share good practice and develop a common
understanding of the research assessment criteria and processes currently utilised
by international funders for the assessment of research and researchers, as well
as the challenges research funders face when implementing these criteria and
processes responsibly.
Conference attendees were invited to fill out a conference feedback survey at the
end of the conference; nearly a quarter of attendees who had actively participated2
in the conference responded.
A set of working definitions
Ahead of the conference, RoRI launched a working paper on The changing role were also made available to
of funders in responsible research assessment: progress, obstacles and the attendees (see Annex A).
way ahead.1 This was developed through a survey of GRC participant members,
combined with desk-research on the state of play. Many of the topics addressed
in the survey of GRC participants were raised throughout the conference. Cross-
referencing to the survey analysis has been included in this conference summary
report. The working paper clearly states that many frameworks on RRA already
exist and that the focus for the GRC’s discussions going forward should be on how
to interpret these in the context of the public funders’ role. This paper acted as
an important springboard and informed discussions at the conference. It was well
received in the community. 55% of the conference feedback survey respondents
said that they found this working paper to be informative and a good document
to use for preparation for the conference and subsequent participation in the
conference discussions.
2. It is to be noted that conference delegates can be referred to as registrants or active participants in this document.
As this was a free virtual conference, anyone could register to attend but not all of those who registered (referred
to registrants) actively participated in the conference. Active participation is defined as participating in at least one
session at the conference.
7
Responsible Research Assessment
Content
43
Approximately 15 hours of plenary
content was available for participants to
watch and participate in, as well as ‘coffee
lounges’, which were informal sessions for
delegates to meet and chat with other
delegates. This diagram provides a
breakdown of the types of content
made available.
Total sessions
38 22 9 7 5
Total sessions Plenary content, Short videos Coffee Closed Regional
available to all including the panel introducing Lounges Sessions (by
participants sessions, keynotes, existing invitation only)
Q&As and regional frameworks
reflections on RRA
Speakers
The conference brought together the perspectives of 45 international speakers from across all 5 GRC regions and
captured many existing practices and approaches to RRA from funders internationally. Throughout the week there was
open discussion on how all funders, regardless of their current engagement on issues of RRA, can improve their approaches
to support a diverse and thriving research culture. Speakers were chosen with diversity in mind. For example, women
represented 56% of the speakers during conference week.3
3. Speakers were suggested by: GRC Gender Working Group, GRC Executive Support Group, the GRC RRA Conference
Advisory Group, the UK Forum for Responsible Research Metrics, and the Declaration on Research Assessment’s
Steering Committee and Advisory Board.
8
GRC Conference Report 2021
Delegates
Recognising that our primary audience - senior management in GRC funding
organisations – would likely be juggling a normal workload alongside attending GRC Region
the virtual conference, the agenda was designed to be as flexible as possible (GRC participant
(e.g. through many on-demand sessions). It is difficult to quantify attendance organisation only)
from senior management from GRC participant organisations given registrants
identified their job titles in a variety of ways, but it is clear from the registrants’
data that a sizeable proportion of registrants were from the targeted audience.
This is also supported by senior management attendance at regional sessions 55 45
(which were by invitation-only).
46
As this is an important topic requiring global partnerships and cooperation, it was
vital for the event to be freely accessible to interested parties. At the end of the 102
conference week, 1078 individuals registered.
The graphs below provide some breakdowns of registrants by GRC participation,
location, organisation type and (self-assessed) topic knowledge. 179
Organisation
427 651
(all delegates)
81 127
100
263
Type of organisation
507
207
352 Funders
Americas
74 Research Performing Organisations Europe
Publisher/Learned society Sub-saharan Africa
Other Asia-Pacific
445
Middle East and North Africa
(MENA)
150 49109
1 (Poor) 2 3 4 5 (Excellent)
377 393
9
Responsible Research Assessment
85%
of the conference feedback
survey respondents agreed
45%
of respondents said that the
conference had significantly
55%
of the respondents
agreed to take forward
that the conference met their increased their knowledge actions arising from
expectations. and understanding of the conference.
RRA.
10
GRC Conference Report 2021
Participation
Looking at active participation in sessions, it was interesting to note
that just under half of the registrants actively participated in the Number of
conference.
attendees
On average delegates participated in 5 sessions during conference participating in
week. The following breakdown provides further detail
25% 1 Session
15 48 13
Americas
Europe
7 Sub-saharan Africa
17 Asia-Pacific
MENA
All figures above only include engagement during conference week. The content
created for and during the conference was available until 25 February 2021 to
those who registered for the conference via the conference platform. Many caught
up on sessions they were unable to watch after the conference week. Between
the end of the conference and the end of the period where registrants had access
to the conference material on the conference platform on February 26, 2021, 60
registrants watched content on the platform; a few seemingly used the time
to catch-up on a lot of sessions. The conference content was thereafter made
publicly available to watch on the GRC YouTube channel.4
4. https://www.youtube.com/channel/UCEuLK_JPExentZ1XA8bGDWw 11
Responsible Research Assessment
Thematic
analysis
The conference agenda was developed
to cover the breadth of the topic and
demonstrate the importance of funders’
role in the research ecosystem. As hoped,
the conference fostered important and
fruitful discussions on the topic.
This section presents an analysis of all the conference content. The key
discussion points were drawn out into the following high-level points:
The analysis reflects the discussions throughout conference week. The discussions
reiterate the existing principles of RRA described in the RoRI working paper1 and
the conference introductory material. The analysis draws out emerging findings
for funders on their role in implementing RRA, including current progress and
challenges. The GRC will use this information to inform future work on this topic.
The information presented as part of this analysis does not currently reflect the
position of the GRC.
12
GRC Conference Report 2021
1
Research
assessment shapes
research culture
Research assessment shapes the research landscape and influences how research is
performed, who it is performed by and how it is disseminated. Research assessment
processes and criteria must incentivise, and recognise, all behaviours that establish
the R&I system that stakeholders would like to foster. What is valued and how it is
measured will drive research culture within and across disciplines, and at national and
international levels.
Research funders are stewards of the R&I system. Funders should take responsibility
and use their power and influence in the research ecosystem to initiate positive change,
leading by example.
Funders must consider how the things they value will translate to what is valued in
the wider research ecosystem. Research assessment approaches that inform funding
allocations map onto individuals and individual behaviours, even where the aim is to
assess teams or organisations. Funders must think about research assessment in the
context of the individual.
While the research system is interdependent (see theme 5), funders are in a crucial
position with more freedom than other stakeholders to initiate change.5 Funders could
move faster and be more radical in in driving RRA and tackling deep systemic issues
but they cannot deliver systemic change alone (see theme 4). However, funders can be
subject to limited resources, lack of sustainable resources, and government policy shifts.
Many university rankings use a small number of proxies of quality (e.g. a narrow set of
metrics) to compare very different research performing organisations. League tables
often do not measure what matters to the R&I system and they do not demonstrate
excellence at a useful scale for the users of this information (for example, pockets of
excellence are not captured).
If a research organisation wishes to ‘climb’ the league table, then they will need
to improve on the indicators of quality which are informing the rank. The values of
organisations are therefore directly influenced. Stakeholders who use or consume the
rankings need to be provided with a much better service.
Funders should:
a.not use existing rankings to inform research assessments
b.not create rankings themselves but instead provide much more
nuanced information in an accessible form
c. take steps to reduce reliance on poorly constructed rankings across
other stakeholder groups.
5. This is due to a number of factors including, but not limited to: funders may have quite a lot of control over the way they allocate
money; many funders can set up their own funding criteria and processes which the R&I sector respond to; funders themselves are not
competing for money in the same ways as individual researchers and research organisations; some funders can consider the health of 13
the whole R&I system especially where they are multidisciplinary funders; funders are not usually ranked like other stakeholders.
Responsible Research Assessment
2
Diverse R&I sectors create high quality
research and impact. This should be
supported in research assessments.
There are challenges with demonstrating research impact due to its wide definitions.
However, defining impact should also be multidimensional and based on context,
discipline and location, as well as accounting for what impact is being assessed
(impact of research, funders or programmes).
14
GRC Conference Report 2021
Funders play a critical role in addressing issues of equality, diversity and inclusion
(EDI) but these issues are systemic and need also to be addressed by universities.
The quality of research relies on and is improved by inclusivity. Funders’
assessment criteria and processes should be inclusive and promote equality
and diversity. Some funders have introduced processes to address bias, gender
being the most prominent characteristic considered thus far. Other protected
characteristics are less addressed but have been identified as representing major
challenges for the future.9
6. See data on how many GRC participants (who responded to the GRC survey on RRA) included gender dimension in
research as part of the assessment (RoRI paper p32)
7. See data on how many GRC participants (who responded to the GRC survey on RRA) currently include open access
publication, open research data, and data curation in their research assessment indicators and how many are
considering including these indicators in the future (RoRI paper p33-35, figure 3)
8. https://wellcome.org/grant-funding/guidance/open-access-guidance/open-access-policy, https://wellcome.org/
grant-funding/guidance/research-organisations-how-implement-responsible-and-fair-approaches-research
9. See data on how many GRC participants (who responded to the GRC survey on RRA) adjusted research assessments
to ensure diversity (RoRI paper p32-33, figure 2)
10. See data on how many GRC participants (who responded to the GRC survey on RRA) adjusted research assessments
to ensure diversity of research (RoRI paper p32-33, figure 2)
15
Responsible Research Assessment
3
Carefully considered and formed
research assessment criteria and
processes foster a healthy R&I system
The excess emphasis (and narrow use of) bibliometrics in research assessment,
especially the focus on journal-level metrics such as Journal Impact Factor (JIF)
and H-Index is particularly prominent in the Global South for hiring, promotion and
funding purposes (in research organisations and research funders). The narrow use of
bibliometrics suppresses diversity of research and researchers.
16
GRC Conference Report 2021
explicit that the long list of criteria is not a checklist, but that only applicable ones are
chosen, are key to delivering RRA.
The most cited contributions which should be recognised and rewarded are
mentoring and wider services to the research community (such as peer review).
Others included teaching, industrial collaborations, technology transfer, societal
impact, and outreach/engagement with publics beyond the academy.12 It follows that
assessments that value the net contribution of an individual to the system - beyond
their direct research contributions - incentivise researchers to play their individual role
in creating a healthy and inclusive research culture.
Research assessment criteria should be clear and transparent, but this should
not undermine diversity. Funders must be transparent, explicit and clear on the
aims and meaning of each criterion (and supporting indicator) they use in research
assessments. For example, when using ‘Research Quality’ as criterion, they should
clarify for both those being assessed and reviewers, what this encompasses and how
it will be indicated.
11. See data on the methods of assessment and the indicators currently used and those being considered for future use
from the GRC survey on RRA (RoRI paper 32-35).
12. See data on which indicators are being included in assessment and those being considered in the future from the GRC
survey on RRA (RoRI paper p33, figure 3).
13. See data on the frequency of internal evaluations to test the robustness of research assessments from the GRC
survey on RRA (RoRI paper p38, table 3).
17
Responsible Research Assessment
Burden arising from the implementation of new assessment methods can create
resistance; resistance from applicants and reviewers needs attention but should
not be a barrier. Funders should explain the changes they make and the problems
which the change is aiming to overcome to get buy-in. It should be noted that
resistance may naturally be countered where there is recognition of a need to correct
imbalances, e.g. in gender. Streamlining processes can also somewhat counter
resistance, as it helps to reduce burden across the whole ecosystem.
Addressing bias and providing equal opportunities are essential to the integrity
of the research assessment process. Funders need to ensure their decision-making
and grant evaluation processes support equal opportunities.15 It is also important
during the review process to examine where bias may have occurred, for example
by checking if a reviewer has conflicts of interest with existing or ongoing projects.
The recent Science Europe study noted that some forms of bias are scrutinised
frequently (e.g. gender 82 %) but others are less often (e.g. ethnicity 31 %).
Addressing unconscious bias, especially in the context of EDI, require funding
agencies to identify risks of exclusion, and develop the appropriate mitigating
actions in research assessment systems.15
14. https://reach.wellcomedatalabs.org/about
15. Technopolis (2019) Science Europe Study on Research Assessment Practices. December 2019. https://www.
scienceeurope.org/our-resources/science-europe-study-on-research-assessment-practices/
16. See data from the GRC survey on RRA about experimentation with new assessment systems and funding allocation
methods (RoRI paper p36-38).
19
Responsible Research Assessment
17. See data from the GRC survey on RRA about the frequency of internal evaluations to test the robustness of research
assessments (RoRI paper p38).
20
GRC Conference Report 2021
4
Global approaches to RRA are pivotal to high-
quality research and impact but they must be
cognisant of local context, culture and language
Some regions rely to a great extent on international peer reviewers (due to limited
local resource with the correct expertise and potential conflicts of interest) and it is
not clear how much local context they have and if this impacts their review.
21
Responsible Research Assessment
5
Funders must steward the system by
acting and exerting the power they hold
to influence other stakeholders
Each stakeholder group’s actions can condition the behaviour of individual researchers
in significant ways and it is up to the whole research community to work together to
achieve cultural change. However, most stakeholder groups shift the blame of the
toxic culture to another party.
Funders can (where possible) take leadership and use their power to kick-start
other changes in the system (see theme 1). It is crucial for funders to understand
how their actions shape the R&I system. This involves a deep engagement and
collaboration with stakeholders (including academia, government, third sector, and
international community) to understand how they work and how funder policies will
impact on how they do their business.
22
GRC Conference Report 2021
It is apparent that all funders do not have a consistent level of delegated authority
from their governments to change criteria and processes for research assessment.
23
Responsible Research Assessment
Regional Perspectives
The tables below include some key points raised in each of the regional meetings.
The full reports can be found at Annex B.
Americas
RRA implementation is diverse across the region, as new approaches are tested
and evaluated in each context.
The definition of RRA includes EDI, however the assessment approaches to best
support and promote EDI in assessment were debated.
RRA and societal value/impact interplay. Mission-oriented research is desired
by funders but the ecosystem challenges this shift (especially where curiosity-
driven research has been the norm).
Assessments should value local delivery and application of research as well as
global impacts.
There is resistance to change research assessments from the academic
community. Cost and complexity to implement novel approaches was a real
barrier for change.
Sub-Saharan Africa
Context is particularly important for the region, especially for emerging
economies who are developing national research ecosystems.
A systems wide approach across the region was desired to strengthen RRA.
Open Science is an emerging policy area for funders in the region. Approaches to
open science differ by agency country or region based on context.
There is a need to strengthen capacities of research performing organisations to
undertake RRA.
The region can share experiences and knowledge about dealing with unconscious
bias, especially in the context of EDI.
Europe
Funders in the region range from conservative to revolutionary in the ways they
assess research.
European funders would like to agree common approaches and methodology for
assessments, especially useful in the context of the European Commission and
funding from Horizon Europe. The region noted the opportunity to collaborate
globally and seek agreement.
Despite efforts from funding agencies indicators like JIF may be used by
reviewers (especially ad-hoc reviewers where practice cannot be called out by
panel members) and embedding widespread reform is needed.
Assessment of interdisciplinary research tends to be conservative especially
where reviewers do not feel expert across the disciplines and where funding is
tight. Similar issues were noted for high-risk, high reward research.
Funders must have a diverse portfolio of uncorrelated risk where individual
programmes have different objectives and therefore assessment mechanisms.
24
GRC Conference Report 2021
Asia-Pacific
Each country in the region reflected differences in how inequity appears in their
systems e.g. geography, gender or ethnicity. Funders in the region have put in
place positive action (otherwise known as affirmative action) for indigenous
people, women and early career researchers.
Where there is not a level playing field for applicants there needs to be provision
of resources for applicants to be able to develop sound and coherent proposals.
Some funders have an emphasis on preventing misconduct and one funder has
put in place greater checks and balances on the misrepresentation of research
and possible fraudulent applications.
Countries within the region have multiple languages and funders need to
support the assessment (including assessors) for parity.
Some funders in the region have a limited number of peer reviewers (which
raises the chances for conflicts of interest), and they rely on international
reviewers who may not understand local context.
A suggested way to improve RRA was to both include clauses to enforce RRA
in contracts and provide expectations in guidance for all research organisations
that might have either funding already in place or for future funding.
There is growing recognition that impact assessment of research funding should
be performed although it is recognised that this takes effort and is expensive.
Funders have used various methods to demonstrate impact.
25
Responsible Research Assessment
Many existing principles on RRA were discussed during the conference, providing
a clear rationale for action. It resonated that the policy landscape is crowded and
further frameworks/principles on RRA are not needed. Instead funders should
focus their efforts on embedding existing principles and taking concrete actions to
incentivise and fulfil RRA ambitions. Any future support from the GRC must factor
in that not all regions and nations are starting from the same point, with diverse
existing implementation of RRA principles and practices.
Funders should think about how to make sustainable and systemic changes. The
GRC could support funders to deliver systemic change by defining what this means
in practice.
26
GRC Conference Report 2021
“
Chair of the GRC Governing Board asked for
”
forum, ensuring that voices from across
the research world are involved.
27
UK Forum for Responsible
Research Metrics