How to go beyond standard metrics for measuring societal impact

Science spending in the UK: Jeremy Hunt’s announcement of a new finance minister to spend 2% of GDP by the year 2025

The finance minister of the UK, Jeremy Hunt, has said that the economic crisis won’t affect ambitious research spending plans.

Universities and scientists were concerned that the government might use funds promised for science to plug the multibillion-pound black hole in the country’s finances caused by interest-rate rises that resulted from budgeting decisions made by the previous prime minister, Liz Truss.

The Russell Group, which represents 24 leading UK research universities, had a sigh of relief at the news the science budget would be protected.

Anne Johnson, president of the Academy of Medical Sciences in London, welcomed the decision in “challenging economic times”. But she warned that there could still be problems ahead for research. “Inflation will continue to put pressure on budgets in real terms, and we must protect collaborations between UK researchers and partners globally.”

The announcement was said to be very good by Paul Nurse who runs the Francis Crick Institute in London.

The new prime minister was a man who served as chancellor in Johnson’s government. It wasn’t known what this change in leadership meant for the future of science spending.

Hunt recommits to spend 2% of GDP on research and development by the year 2025. The UK Office for National Statistics says the government has met the target. This can be seen due to the changes in how research and development spending is calculated.

The website for the Department of Business, Energy and Industrial Strategy (BEIS), which includes the science spending, states that both Freeman and Ghani areministers of state. The spokespeople for BEIS stated that the ministers portfolios aren’t officially confirmed and that the overall responsibility for science wasn’t known.

The Nature Index: Count and Share for Tracking Research Outputs of Countries/Territors/Instituencies, from 2000 to 2020

A description of the terminology and methodology used in this supplement, and a guide to the functionality that is available free online at natureindex.com.

Count and Share are used to track research output. Each article that has at least one author from that country/territory or institution will get a Count of 1 for it. This is the case regardless of the number of authors an article has, and it means that the same article can contribute to the Count of multiple countries/territories or institutions.

There is a small variation in the number of articles in the Nature Index journals. The percentage difference in the total number of articles in the Index in a given year relative to the number of articles in a base year is used to arrive at it.

The bilateral collaboration score (CS) between two institutions A+B is the sum of each of their Shares on the papers to which both have contributed. A bilateral collaboration can be between any two institutions or countries/territories co-authoring at least one article in the journals tracked by the Nature Index.

Each query will return a profile page that lists the country or institution’s recent outputs, from which it is possible to drill down for more information. A journal can be used to display articles. Research outputs are organized by subject area. The pages list the institution or country’s/territory’s top collaborators, as well as its relationship with other organizations. Users can track an institution’s performance over time, create their own indexes and export table data.

For each of the leading 500 institutions in the Nature Index, its scholarly works between 2000 and 2020 (Scholarly works 2000–20) were queried from The Lens database. All the Lens patents that cite its scholarly works were also uniquely counted. The institution’s influence on patents is measured by the total number of scholarly works and the unique count of citing patents. The patent influence metric was used to rank the 500 institutions. Academic, government and NPO/NGO institutions were ranked separately by the metric for each sector from the leading 500. There were no subsidiary research institutions in those sector rankings.

The percentage change in biomedical research Share in the Nature Index in some leading countries saw distinctly different trends during the first two years of the COVID-19 crisis. A number of nations had a significant increase in Share from the year 2000 to the year 2020. Such a boost could have contributed to a percentage fall for some nations the following year, as research volumes fell back to levels closer to those seen in pre-pandemic times. In the year 2021, China and Israel saw a rise in Share, while India dropped in 2020 but bounced back the following year.

Harvard University again features among the top academic–corporate biomedical partnerships in the Nature Index by bilateral collaboration score, but the top collaboration involves two Switzerland-based organizations, the University of Basel and international pharmaceutical company Novartis.

Other research2 has suggested that scientific innovation has slowed in recent decades, too, says Yian Yin, also a computational social scientist at Northwestern. He says that the new start to a data-driven way to investigate how science changes is offered by this study.

Walsh says that it is important to understand why the changes are so drastic. Changes in the scientific enterprise may be part of the trend. For example, there are now many more researchers than in the 1940s, which has created a more competitive environment and raised the stakes to publish research and seek patents. The incentives for researchers to do their work has changed because of that. Large research teams, for example, have become more common, and Wang and his colleagues have found3 that big teams are more likely to produce incremental than disruptive science.

The authors found that in the 1950s, research was more likely to use words such as creation or discovery, but in the 2010s, it was more likely to use terms like incremental progress.

“It’s great to see this [phenomenon] documented in such a meticulous manner,” says Dashun Wang, a computational social scientist at Northwestern University in Evanston, Illinois, who studies disruptiveness in science. “They look at this in 100 different ways, and I find it very convincing overall.”

Disruptiveness is not necessarily good, says Wang. The first direct observation of gravitational waves, for example, was both revolutionary and the product of incremental science, he says.

The ideal is a healthy mix of incremental and disruptive research, says John Walsh, a specialist in science and technology policy at the Georgia Institute of Technology in Atlanta. He thinks it is a good idea to have more reproduction and replication in a world where we are concerned about validity.

Finding an explanation for the decline won’t be easy, Walsh says. Although the proportion of disruptive research dropped significantly between 1945 and 2010, the number of highly-disruptive studies has remained about the same. CD indices fell steeply from 1945 to 1970, then fell more gradually from the late 1990s to 2010; this is puzzling. In the 2000s, he says disruptiveness needs an explanation for levelling off.

Nightingale thinks that there is a good reason to try to evaluate impact and that the return on investment would be high. Efforts over the past few years by Bornmann, Meijer and others are laying the groundwork for a fuller evaluation of science and the effects it has on policy, the economy and other areas.

In 2016, the Netherlands Association of Universities of Applied Sciences (NAUAS) took a broad view of scientific contributions in a document called Onderzoek met Impact (Research with Impact) (see go.nature.com/3wtq3). The report says that the research should affect a number of areas, including health and vibrancy, education and talent development, societal resilience, sustainable transportation, and sustainable agriculture. The document shows the NAUAS interested in proving the value of research, but it doesn’t include a way to track impact.

It is unlikely that a score like this could be used to identify the scientists who have changed the actions of governments, medical societies or other institutions. They compared policy-citation scores from different universities with their impact database scores. Their analysis suggested that institutions with the most policy citations tended to have been given higher scores by peer reviewers in the REF, but the correlation wasn’t especially strong. One take-home lesson: a simple score derived from a policy database is likely to fall short, Szomszor says. Including some form of peer review “that would allow you to judge the worth of policy influence” would be preferable.

At an even wider level, science has the potential to transform the planet by tackling global environmental crises, such as climate change, and to improve the well-being of poorer communities. The UN Sustainable Development Goals, which address the biggest global challenges including poverty, hunger and clean energy, are now being used to measure impact.

Bornmann co-authored a 2020 study5 that examined altmetric scores — figures based on social-media posts, news mentions and Wikipedia entries, as gathered by the data company Altmetric — of about 200 researchers who applied for funding at the Competence Center Environment and Sustainability, an environmental research centre based at the Swiss Federal Institute of Technology in Zurich. The study found 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299 888-353-1299

It is a bad situation if alternative metrics are being used, even if it is not clear whether they are valid, says Bornmann, who adds that it is a bad situation if such metrics are being used. “There needs to be a longer period of research on these metrics, and then they can be used in evaluation,” he says.