Peer Review - before you submit

It can be extremely frustrating to read reviewer comments on your grant application when you have been turned down for funding. But, one way to alleviate surprises that may show up in reviewer comments is to engage a network of reviewers before you submit. Below are some recommendations for selecting and recruiting internal reviewers.

Aim for three
ORDE suggests that PIs try to find at least three people to review their grants before submitting. This gives you a nice diversity of perspectives and ups your chances that you'll hear some feedback that will allow you to revise your application in such a way that you don't hear that same feedback in your agency review.

Choose the right level of reviewer expertise
Certainly, you want to have people with the right expertise to emulate those who will review your grant at an agency. Look for folks in your field who will be familiar with the concepts you're using, but not those who are so close to the project (e.g., Co PIs) that they may overlook some of the same problems in your application that you've overlooked.

Find a layperson
We've heard several seasoned PIs say that they have a family member (a grandma, a daughter, a spouse, etc.) review their proposal before submitting. These generous family members are able to give feedback on whether your argument is clear, if you've adequately described the need for your research, and whether the proposal includes too much jargon. This is a useful perspective, especially if the agency you're submitting to may include lay people as reviewers.

Give direction
I've found that at times when I ask a colleague to review my writing, if I'm not clear with them on what I want, they sometimes go through and simply identify grammatical and/or spelling errors. This is certainly helpful, but not until I have a fairly final product. But, when folks don't know what to look for, they often focus on the little things. So, give your reviewers direction. Depending on how busy they are, you may want to send one piece of your proposal to a colleague or friend and have them assess one aspect, such as whether you've shown the potential impact of your project clearly and compellingly.

Give time and a heads up
Those in the grants game know that even when you start early, time is always of the essence when you're developing an application. Yet, if you want to adequately use internal reviewers before you submit, and you want them to continue talking to you, it's important to give them time. Even before you have something to review, it's a good idea to identify those you want to review your application and ask them if they're willing. Then let them know when to expect a draft and when you need it back (giving them a week is usually appropriate). Lastly, make sure that you have enough time to really use their feedback. If they give you comments, you need to make sure you have time to incorporate them in a revision, otherwise it's all for naught.

Resources
ORDE can help! If you are faculty member for CU Denver | Anschutz Medical Campus, Associate Director, Naomi Nishi, will review your grant overview and project description from a lay perspective, if you get it to her at least a week before routing through OGC. Email her to set it up.

The dangers of submitting at the last minute

You have 15 minutes before your 5:00 pm grant application deadline. Your weeks of late nights getting your application together is about to pay off.  You hover over your grants administrator's shoulder as they click the final button to submit and you both look on with bated breath for a confirmation message. Instead of the anticipated confirmation, you get... an error message! You wake up in a cold sweat. Whew! It was just a dream. You submitted your final grant application yesterday, a full week before the deadline. You're good!

Unfortunately, this nightmare or one like it sometimes ends up being a reality for PIs that wait till the last minute to submit grants. Now we know full well that oftentimes PIs do not get the sort of lead time they need to really take the time necessary. ORDE recommends that PIs take a 6 month time frame to develop their idea and project to submitting the grant. But we also recommend that your zero hour not be at the grant deadline. Here's why:

Error messages: As our nightmare scenario suggests, some submission systems will actually check for completion of your grant application and will not accept it until all errors are corrected. The NIH only considers an application if it is accepted as error free by grants.gov and then the NIH. If you get an error message, you must start the submission process over again with errors fixed.

Site overload: Unfortunately, most PIs wait till the eleventh hour to submit their proposal, and so the agency site is sometimes experiencing hundreds or even thousands of submissions at a time.  This means that processing can slow down or even that sites can crash. Sure, if the agency site crashes, they may make exceptions, but do you really want to be at the mercy of that call when you and your team have done so much work to get that application together?

Missing pieces: Over half of applications are not reviewed because they do not follow directions or the project they're proposing is not appropriate for the agency. Obviously, you don't want to be in that pool, but when Program Announcements and RFPs have pages and pages of directions, it's easy to miss something when you're scrambling to complete your final application right at deadline.



Processing: As described above, the PI is not the person who hits the final submission button. There is a whole team of people working with you on your grant to get it submission-ready. You need to work with your grants administrator and the Office of Grants & Contracts to submit an application. Waiting until the last minute can put your whole team in a tight spot.

Anticipating the unanticipated: One thing all of these reasons have in common is they're often not anticipated by the PI. Sure, your application would have been on time if this or that hadn't happened, but of course in today's competitive climate, it's becoming rarer for agencies to make exceptions to missed deadlines. So, save yourself the heartache and stress... submit early!


Resources:
ORDE timeline
What you need to know about receipt and referral - NIH Video

NIH Scoring Glossary

I was feeling ambitious today, and thought I'd focus our blog on the NIH's scoring system, and the corresponding numbers PI's are sometimes given to decipher the likelihood of being funded before they actually know. To make it easy, I thought I'd set it up as a glossary.

Scores
Once your application is received at the NIH, it is assigned to a study section, and the Scientific Review Officer (SRO) who manages that section assigns your application to reviewers. Those reviewers assign your application a score on a 1-9 scale. 1=exceptional and 9=poor. They assign a score for each criteria: significance, investigator(s), innovation, approach, and environment, as well as a preliminary overall impact score. Those applications that receive the best or lowest preliminary scores for overall impact are discussed at study section, and every reviewer (not just those assigned to your application) submits final scores for all criteria. The mean of these scores is taken, it is multiplied by 10, and this final figure is your review score and will fall between 10 and 90 with 10 being a perfect score.

Percentiles
Some study sections will provide a percentile along with a score on a PI's application.  This percentile is the percentage of applications the study section has reviewed in the past year that received a better score than yours. Thus, if you are in the 5th percentile, your application has scored better than 95% of applications reviewed that year. These percentiles are given so that applications may be ranked across the NIH; it weights them in a sense. Some study sections are notorious for being really hard graders and others more regularly assign excellent or even perfect scores, so to be equitable, by looking at percentiles, we get a better sense of the comparative merit of proposals.


Paylines
Now, the previous definitions have shown you the significance of those numbers, but you're probably thinking what really matters is getting the funding! Paylines take us closer to that decision. Once your application has been scored and possibly given a percentile (this is not always given to the PI), it goes to the Institute/Center (IC), where an advisory council/board reviews them. The IC Director, who is ultimately responsible for deciding which projects get funded, working with their* council, looks at their budget and the applications and sets a payline, above which they fund almost all applications. Now, the Director can decide to fund applications that fall below the payline, but that are important to their mission or the institute's current focus. For some institutes, this is not cut and dry, some directors choose from the top scoring applications, which they want to fund and ultimately report how many applications were funded at different percentiles.

Even though it seems like we should be able to know if we're going to be funded once we have a score, percentile, and payline, the truth is that there is no certainty. Until the Director makes the final decision to fund your proposal, it's speculative. This may seem maddening, but remember that the Director is looking to fund not only the best projects, but the best projects for them. They want to fund those projects that will best move their* IC forward, so this can work to your benefit when you are careful to align your research with the goals and needs of a particular NIH IC.

*If you think this is a misuse of "their," please see last week's blog to find out why you're wrong! :)

Resources:
Paylines, Percentiles, and Success Rates - NIH Blog: Rock Talk
Scoring - NIH Description of Scoring

Editing and Usage: Times are a-changin

 
With a new year comes big news on the grammar and editing front! The style guide for The Washington Post in now accepting the word: they to refer to a singular noun. This of course led to cheering and some uproar in the editing and the language police communities. I was thrilled by the change, and as Mary Norris, copy editor for The New Yorker, suggests in her new book, felt it was inevitable.

For those of you who are not nerds in this area, this just means that according to the Washington Post Style Guide, it is now okay to say "One should be careful with their words, shouldn't they?" I'm using their and they to refer to the singular one. Under the old, nit-picky rules, I should have said, "One should be careful with his or her words, shouldn't he or she?" Even the sticklers out there would agree with me that the first example is better, right?

But, let me bring it back to why it's relevant to grant writing... It's not. If the creators of the Washington Post Style Guide were reviewing your grant proposals, this would be important to remember. But, they're not. And, the first rule of grant-writing is Thou shalt write for thy audience. So, when you're writing your grant proposal, consider the norms in your field. Stick to common usage unless there are specifics to your field. For instance, use the word use instead of utilize. Simplify where you can and make all language and editing decisions with your likely reviewers in mind.

Happy New Year!

Resources
The Washington Post Style Guide Now Accepts 'Singular' They - mental_floss blog
OWL: Online Purdue Writing Lab - Purdue University