The grant review will be changed to reduce bias

Changing the Peer Review System to Prevent Bias and Improve Access to Research Resources in a Big Funder: Comment on the NNIH Proposal

The US National Institutes of Health (NIH) has released a tentative plan to change how its research grant applications are scored, with the aim of reducing bias and lowering the burden on reviewers. The reviewers would not rate researchers’ expertise or their institutions’ access to resources, under the new system.

In addition to considering the race and gender of reviewers, ensuring that panels have representation from under-represented institutions could have a big impact, says Enrique Neblett, a public-health researcher at the University of Michigan. TheNIH needs to make a bolder move given the persistence of funding disparity.

Researchers who spoke to Nature say that changes to the peer-review system are long overdue, and that the proposals are a good start. But some think the revisions don’t go far enough to correct these entrenched problems.

The last category, which will be scored under the proposal, is frequently misinterpreted. Reviewers sometimes score applicants without considering them as part of the proposed research. This has led to higher scores for prestigious institutions and individuals. Under the proposal, rather than score this category, reviewers would choose whether they think researcher expertise or institutional resources are adequate or not. They can leave feedback in a text box on the review form if they select the latter. This will “prevent reviewers from waxing poetic about a really famous investigator that tilts the evaluation of the science”, Byrnes says.

These changes are bold for a big funder like the National Institute of Health, according to a research policy specialist at EMBO in Heidelberg, Germany.

Some advisers attending the 8 December meeting pushed back on the plan, suggesting that researchers and institutional resources are crucial factors in determining the merit of research projects. According to a University of Pennsylvania epigeneticist, there is value in assigning an objective score to an investigator. Without a score, Berger added, it could be difficult to understand the reviewer’s thinking and how they factored researcher expertise into their decision. The reviewers can leave comments about their concerns, which could be used to evaluate the overall impact score.

The CSR is accepting public feedback about the revisions until 10 March 2023. It will also release proposed changes to the criteria for scoring clinical-trial and training grants in the coming months.

The National Science Foundation is considering a funding model which would allow grant peer reviewers to veto the decisions of their colleagues once in a funding cycle. Peer reviewers often disagree on what grants should be funded. In theory, the golden-ticket model would mean that unconventional, high-risk ideas that do not have the unanimous backing of all referees would end up being funded.

Every researcher has the same feeling; their proposals are innovative and outside of the box, but the reviewers don’t get it.

There are some golden-ticket schemes in Europe. In 2017, the Villum Foundation, one of the largest philanthropic funders in Denmark, committed to allocating 15% of its annual funding to ‘risky’ grants that applicants might not otherwise consider submitting. In 2022, 51 researchers received 100 million kroner (US$14.3 million) through the scheme.

The Villum Foundation also made sure that reviewer panels don’t meet to discuss proposals, Sinkjær notes. After rolling out the golden-ticket scheme, the foundation found that there was no correlation between success in securing grants and the career stage of applicants. “You’re as successful as a postdoc when applying to this programme as you are as a full professor,” Sinkjær notes.

But Heidi Williams, an economist at Stanford University in California who specializes in science policy, says that blinding proposals could exacerbate disparities in science. She notes, for instance, that experienced academics might write better grant proposals than do early-career colleagues, and that blinding the reviewing process means researchers from marginalized groups can’t be given extra support.

Golden tickets are one way the United States is trying to promote high risk, high reward science. The Advanced Research Projects Agency for Health (ARPA-H) was created by the Biden administration in March of last year. The ARPA-E, which focuses on developing new energy technologies, is the same type of research as IARPA.

Sinkjr says that if you want to go for truly innovative research ideas you need to focus on the research idea and not solely on the person proposing it.