How can we measure research impact? Funders share their insights

Research can greatly benefit society: today’s researchers are tackling some of the world’s greatest challenges, including climate change and COVID-19. This kind of research is not unidirectional; while adding to our collective knowledge, it’s shaped by society’s needs, unfolding global events, discussions with policymakers and communication with the public.

Funding organizations have long understood the importance of research with an impact on society. With increasingly stretched budgets, researchers are called on to show the economic and societal impact of their work, and funders have systems in place to evaluate this. Yet demonstrating the impact of research remains a challenge for everyone involved. Why is it so hard to measure? And what can we do about it?

In October and November 2021, Elsevier convened a series of workshops on “Calibrating Economic and Societal Impact: Best Practices to Inform Ex-post Evaluation” with a group of similarly sized funders in research-intensive economies. Leading researchers and managers from The Academy of Finland opens in new tab/window, Science Foundation Ireland opens in new tab/window, Israel Science Foundation opens in new tab/window and New Zealand’s Ministry of Business, Innovation and Employment opens in new tab/window participated in this series. Additional expert input to inform the discussion was contributed by Elsevier’s International Center for the Study of Research (ICSR).

Four themes emerged

Participants found common ground: they shared many challenges related to demonstrating the impact of research, and their discussions about how to improve it centered around four main themes that lean toward tackling important but answerable questions.

1. Linking research excellence and societal impact

Researchers, institutions, funders and others generally consider excellence a necessary element for research to have a positive impact. As a result, many assume that when research is funded and carried out to a high standard, it will have an impact.

However, our understanding of the link between research excellence and the impact of the research is actually quite limited. There is evidence that higher quality research proposals are linked to research results that have a greater positive impact on the advancement of knowledge. But when it comes to evidence of the link between research excellence and societal impact, we have limited data showing a weak correlation opens in new tab/window. Building our knowledge of the link would help improve the way impact is measured and demonstrated.

Questions the workshop participants proposed:

• Identifying a causal relationship may be out of reach, yet as more data becomes available, can the community develop its understanding of the long-term benefits of funding research excellence? To what extent is that effected by the chosen field of study?

• If research excellence helps drive societal impact, does that preclude non-excellent research from the same results? Exploring that would broaden our understanding of the link between quality and impact.

2. Communicating research for impact

Regardless of its excellence, research can only make an impact if the right people know about it. This makes communicating relevant research a vital step in creating and demonstrating impact. But it’s not as simple as it may seem to determine which research is relevant. While research that is focused on real-world challenges like COVID-19 and ocean plastic is considered most likely to have an impact, there are many examples of research that has led unintentionally to groundbreaking results. Furthermore, it is now well established that the research-impact relationship isn’t a linear one.

With this in mind, even when the research was not intended to solve a problem, it’s important to communicate it publicly. In cases where there is a clear intention to tackle a societal challenge, researchers can best make a demonstrable impact by connecting with people throughout the process – particularly those affected by the challenge and those who could turn their results into real-world action. Several different people might be involved in this communication process, including the researchers themselves and institutional partners like press officers and technology transfer officers.

Questions the workshop participants proposed:

• To bring greater understanding to the contextual factors that drive societal impact or relevance, which interventions are most effective? What can funding organizations, research institutions and researchers do to ensure the impact is realized?

• This could be extended to a broader look at the effect the development of research-adjacent activities like press outreach has on societal impact. Has the focus on societal challenges made a difference in the form of an acceleration of impact?

3. Understanding the impact of research-based education

It’s often considered most effective to track and measure the overall impact of a research program, portfolio or institution rather than the individuals conducting the research. Yet those people are a conduit for knowledge: wherever they go in their career, they take their skills, experience and contacts with them.

A research-based education, such as a PhD, does not always result in a career in academia; they often move into governmental and private sector jobs. People whose PhDs were funded can make an impact through non-research careers and by collaborating with former colleagues and advisors in their positions in industry, for example. Understanding the paths people take in their careers can help funders and other organizations track the societal impact of their scholarships more accurately.

Questions the workshop participants proposed:

• Could the best data sources, partners and methodologies be identified to ensure more of the most insightful studies can identify job titles or roles, new companies founded, and physical location?

• What are the best approaches to capturing the full extent of societal impact of those with a research-based education?

• There was also a strong desire to be able to move beyond one-off snapshots and move to a continuous monitoring approach: is that possible?

• What impact results from the connections built through the movement of people? Do those that move into industry maintain partnerships with those from their former research institution and do those relationships inform their work and societal impact?

4. Determining the costs and benefits of impact assessments

Many countries now have performance-based research funding systems (PRFS), which rely on assessments of research excellence and impact. These can provide significant benefits beyond the purpose of the assessment: in addition to boosting an institution’s reputation and therefore attracting more funding and collaboration, the results also provide valuable data for researchers studying research itself.

Yet this kind of assessment can be a significant burden on research institutions and researchers themselves, who invest in preparing information. Financial and time costs are common challenges associated with processes like the Research Excellence Framework (REF) opens in new tab/window in the UK. Some also question the usefulness of impact statements and other elements of the assessment, as the complexity of the link between research and societal impact becomes clearer.

Questions the workshop participants proposed took these issues into consideration:

• There is a need to keep reviewing the effectiveness of the various assessment exercises. Which approaches are most effective?

• If resourcing and financial costs prevent measuring the excellence and impact of all research, what is getting missed?

They suggested that a comprehensive understanding of the burden assessment places on the research community might also benefit the research community.

Developing research impact studies

Throughout their discussions, the workshop participants were keen to prioritize tangible questions around challenges the whole research community is facing — questions that will directly inform how funding organizations think about impact. They identified two potential research projects:

• Identifying success factors for impact: Testing the assumption that societal impact of research is more likely to occur where research excellence is coupled with societal relevance and contextual factors that contribute to the realization of impact.

• Societal impact of research-based education and training: Testing the assumption that societal impact of research can be created by people who leave research and pursue careers that make use of their research-based education and training.

Elsevier’s International Center for the Study of Research (ICSR) is organizing “deep dive” sessions together with interested workshop participants to begin framing the research questions. Following this, the ISCR will develop either or both of the two project ideas together with collaborators to assess their feasibility for a proof-of-concept study.

Explore some of the resources shared during the workshops:

• 2019 GRC Statement of Principles opens in new tab/window

• Bibliometrics: The Leiden Manifesto for research metrics opens in new tab/window

• Tertiary Education Commission Performance-Based Research Fund opens in new tab/window

• Broadening the Scope of Impact, 2015, Small Advanced Economies Initiative opens in new tab/window

• Beyond Academia – Interrogating Research Impact in the Research Excellence Framework opens in new tab/window

• Rethinking research impact through principles for fair and equitable partnerships opens in new tab/window

• Aligning scientific impact and societal relevance: the roles of academic engagement and interdisciplinary research opens in new tab/window

• Between relevance and excellence? Research impact agenda and the production of policy knowledge opens in new tab/window

• UBC PhD Career Outcomes opens in new tab/window

• What is known about doctoral employment? Reflections from a UK study and directions for future research opens in new tab/window