NIH Scoring Glossary

I was feeling ambitious today, and thought I'd focus our blog on the NIH's scoring system, and the corresponding numbers PI's are sometimes given to decipher the likelihood of being funded before they actually know. To make it easy, I thought I'd set it up as a glossary.

Scores
Once your application is received at the NIH, it is assigned to a study section, and the Scientific Review Officer (SRO) who manages that section assigns your application to reviewers. Those reviewers assign your application a score on a 1-9 scale. 1=exceptional and 9=poor. They assign a score for each criteria: significance, investigator(s), innovation, approach, and environment, as well as a preliminary overall impact score. Those applications that receive the best or lowest preliminary scores for overall impact are discussed at study section, and every reviewer (not just those assigned to your application) submits final scores for all criteria. The mean of these scores is taken, it is multiplied by 10, and this final figure is your review score and will fall between 10 and 90 with 10 being a perfect score.

Percentiles
Some study sections will provide a percentile along with a score on a PI's application.  This percentile is the percentage of applications the study section has reviewed in the past year that received a better score than yours. Thus, if you are in the 5th percentile, your application has scored better than 95% of applications reviewed that year. These percentiles are given so that applications may be ranked across the NIH; it weights them in a sense. Some study sections are notorious for being really hard graders and others more regularly assign excellent or even perfect scores, so to be equitable, by looking at percentiles, we get a better sense of the comparative merit of proposals.


Paylines
Now, the previous definitions have shown you the significance of those numbers, but you're probably thinking what really matters is getting the funding! Paylines take us closer to that decision. Once your application has been scored and possibly given a percentile (this is not always given to the PI), it goes to the Institute/Center (IC), where an advisory council/board reviews them. The IC Director, who is ultimately responsible for deciding which projects get funded, working with their* council, looks at their budget and the applications and sets a payline, above which they fund almost all applications. Now, the Director can decide to fund applications that fall below the payline, but that are important to their mission or the institute's current focus. For some institutes, this is not cut and dry, some directors choose from the top scoring applications, which they want to fund and ultimately report how many applications were funded at different percentiles.

Even though it seems like we should be able to know if we're going to be funded once we have a score, percentile, and payline, the truth is that there is no certainty. Until the Director makes the final decision to fund your proposal, it's speculative. This may seem maddening, but remember that the Director is looking to fund not only the best projects, but the best projects for them. They want to fund those projects that will best move their* IC forward, so this can work to your benefit when you are careful to align your research with the goals and needs of a particular NIH IC.

*If you think this is a misuse of "their," please see last week's blog to find out why you're wrong! :)

Resources:
Paylines, Percentiles, and Success Rates - NIH Blog: Rock Talk
Scoring - NIH Description of Scoring