Friday, June 24, 2016

NIH Updates: Rigor and Reproducibility

You've probably heard the discussion around NIH projects and the questions around rigor and responsibility taking place in the last few years. But what does this mean for your next grant application?

The updates around rigor and reproducibility focus in four areas: premise, design, variables, and authentication.

Premise: Sure, PIs have always needed to show preliminary data or results when applying to the NIH, but now you need to take it a bit further. In addition to discussing your preliminary work, you must also discuss the strengths and weaknesses of it. You must now vet the foundation of your project in your grant application.

Design: The NIH is expecting more rigorous experimental design; their definition of rigorous includes the use of methods that are reproducible. As they state, "Robust results are obtained using methods designed to avoid bias and can be reproduced under well-controlled and reported experimental conditions." (NIH website, 2016)

Variables: A couple years ago, Sally Rocky, former Director of Extramural Research at the NIH, reported that a good majority of projects funded by the NIH did not include female animal models. In response to this particular overarching bias, the NIH now expects that sex as well as other relevant variables be accounted for in project design in funded grant applications.

Authentication: According to Hughes (2007), "between 18-36% of cell lines might be misidentified or cross-contaminated." Thus the NIH "expects that key biological and/or chemical resources will be regularly authenticated to ensure their identity and validity for use in the proposed studies" (NIH website, 2016). For AMC researchers, the Barbara Davis Center's Molecular Biology Service Center
does offer authentication services.

The NIH put the following diagram together to look at the various reasons and implications for their new rigor and reproducibility. The link and other resources are available below.

NIH New Grant Guidelines Diagram
Updated Application Instructions
Presentation from the Department of Medicine - Jenny Kemp, PhD

Monday, June 20, 2016

New investigator grants

ORDE has spent the last several weeks updating our New Investigator e-book, which is just out. I thought I would focus the blog on some things to consider when looking for a new investigator grant program. Below are four questions you'll want to answer before applying to any new investigator grant program.

How does the agency define new investigator?
If you're a new investigator, you're a new investigator, right? Well, maybe. Different agencies define new investigator differently. Some are looking at how many years since you received your terminal degree(s). Some are looking at how long you've been in your research position. And, some are looking at whether or not you've received major funding previously.

Is it a mentored grant?
Some new investigator programs are the same sort of research-focused programs as those that are not for new investigators. And others are considered mentored awards, where in your proposal you must address your own career development plan, and also identify a mentor who will work with you throughout the award period. The NIH Career Development or K awards are generally framed in this way. While K applicants must identify a research project in their proposal, the larger focus is on the candidate, their mentor, and their career development. On the other hand, the NSF's CAREER program is research-focused. While applicants are wise to show how their CAREER project fits in with their and their department's larger research goals, this is peripheral to the research project itself. Other agencies run the gamut.

Do your past grants affect eligibility?
At some agencies, the new investigator programs are targeted at bringing very early career investigators and their research up to speed. Thus, if you have shown that you are competitive for major funding previously, this could make you ineligible for some new investigator awards. For instance, at the NIH, if you have secured major funding as the PI, e.g., received an R01, you lose your new investigator status and would not be a good candidate for a K award. But, for the NSF CAREER program, about half  of CAREER awardees have received previous awards from the NSF and it puts them in a better place to compete for the CAREER and certainly does not make them ineligible.

What are the goals of the program?
The questions above really all lead to this question. Before you decide whether or not to apply for a new investigator grant, you must first understand the goals of the agency and the new investigator program. Is the agency hoping to create new independent investigators with their program by funding career development? Or is the agency looking to promote those newer investigators who have already proven that they are independent and productive researchers? When you understand the program, you can consider if it is a good fit for you at your current stage.

New Investigator e-book - ORDE
ORDE Funding resources

Wednesday, June 8, 2016

The proposal abstract: Sell your idea early

Last week I attended the annual research development conference (NORDP), and had a chance to hear from my favorite grant coach, Robert Porter, PhD. Dr. Porter was touting the importance of the grant abstract for selling an idea to reviewers at the outset. More specifically, for NIH applicants, he was referring to the Specific Aims and for NSF applicants, the Project Summary. He made the point that when reading grants, reviewers make up their minds about a proposal in the first two paragraphs or at least in the first couple pages. Thus, it is essential that your core argument fall in the opening of your proposal.

Dr. Porter recommended the following three paragraph formula for writing your abstract:

Paragraph 1: Lay out your problem
  1. Grab the reader's interest
  2. Explain why it's important
  3. Summarize the state of the art and limitations
  4. Describe challenges to solving the problem
You want to begin your proposal by driving home the urgency for your project. Explaining what's wrong, what needs to be done, and setting up the importance and need for your research is essential. Porter suggests that you use words like "exciting," or "unprecedented" to convey confidence. However, I find that this language often comes across as hyperbole. It's always better to show how your project is exciting or unprecedented instead of just saying it is.

Paragraph 2: State your solution
  1. Describe your concept and credibility
  2. Describe the project's fundamental purpose
This is the section where you want to put the meat on the bones of your argument. You've communicated the need and excitement for your project, now you have to give your reviewers confidence that you can actually deliver something substantial.

Paragraph 3: Create a vision

  1. Show how your work will advance the field
  2.  Envision the world with the problem solved
This is really the icing on the cake that directly relates your vision for your project with that of the agency to which you're applying. Show the big picture of your project; let them know that this is the just the exciting start to even larger results. But, at the same time, never forget that your agency also has a vision and a plan for what they want to do. Your vision should be aligned perfectly with theirs and you should say how they are aligned directly.

Crafting a Sales Pitch for Your Grant Proposal - Robert Porter
Six Critical Questions to Launch a Successful Grant Proposal - Robert Porter

Friday, May 20, 2016

Getting and giving peer feedback

Peer feedback is really a crucial tool for faculty researchers. It is core to the review processes for publications and for grant proposals. But even beyond this, peer feedback is essential for getting your manuscript or your grant proposal into shape before you submit. However, giving and receiving feedback is oftentimes an overlooked skill, but one that can make all the difference for you and your colleagues when employed well.

At ORDE, we recommend that PIs get three internal reviews of their grant and integrate the feedback before submitting. One of your reviewers should be a layperson and the other two should be experts in your field, but not familiar with your project (you don't want your peer reviewer intuiting things in your proposal that aren't there; you want them to identify what's missing).

Getting feedback
When asking for feedback from your peers, be specific with the kind of feedback you want. I've found that when asking for feedback on my writing, if I don't explain what sort of feedback I'm looking for, my reviewers tend to resort to their grammar school training and focus on punctuation and spelling rather than advising me on clarity.

Along with giving your reviewers instructions on what you want them to look for in your proposal, also give them enough time to adequately accomplish the task. If you're waiting till the last minute to pull things together, and then dropping it on your colleague's desk, not only will you frustrate that colleague and likely get sub-par feedback, but you probably won't have time to really use the feedback they do give you. On that same point, give your colleague a heads up. Ask them if they can review your proposal weeks in advance, tell them when you will send it to them, and agree on when they can have feedback back to you.

Lastly, be sure to thank your peer(s) for investing their time in you and your proposal. Make sure you offer to return the favor and let them know how important their feedback was to you. And, circle back around when you get that grant to thank them for their role in your success.

Giving feedback
When your colleague comes back around to ask you to review her proposal, some of the same tips apply. First of all, ask your colleague to plan ahead to ensure you have adequate time to review the proposal and she has adequate time to incorporate useful suggestions. Also, ask your colleague what she wants you to look for. Is she still tweaking her idea? Is she open to methodological suggestions? Does she want feedback on clarity or persuasiveness? If she asks you to proof it, you may want to suggest that they find an editor for that.

Giving feedback doesn't seem like a difficult concept, but many folks are uncomfortable with giving constructive feedback to a friend or colleague. You don't want to hurt any one's feelings after all. I actually think the reason people stick to grammar and spelling feedback is because they're nervous about offending a colleague. Yet, constructive feedback is essential to get better. However, good editors or reviewers understand how sensitive people are about their work. The editing adage, "murder your darlings" is often used, because getting feedback on your latest pride and joy can feel quite personal.

So, to deliver your necessary but perhaps sensitive feedback, some suggest a sandwich approach. Begin by telling your colleague everything you liked about their proposal, then move into some constructive critique, and end with more positive. In using this approach, it's important to be sincere in what you found as strengths. And, instead of dividing up the positive attributes, share all of them at the front end and finish by describing how excited you are about their project.

Also important to giving good feedback is being specific with your critique, and making sure your advice is usable. We don't call it constructive criticism for nothing. Negative feedback that is not useful or can't be applied is just mean.

I included a discussion of getting and giving feedback, because to be successful, you need to do both. If you make it a habit to ask for feedback but never find time to give it, then it won't belong before your peers are too busy to look at your proposal. Reciprocity is the name of the peer feedback game.

Giving and Receiving Feedback: A Guide to the Use of Peers in Self Assessment - UTS
The Delicate Art of Giving Feedback - Harvard Business Review

Friday, May 13, 2016

Developing your writing practice

I recently heard one of our faculty members compare writing to exercising. In that, they're both hard to do either one consistently. But both are good for you and you need to practice them to develop your skills. She warned that, like shirking your daily run when planning to run a marathon, you'll be in a lot of pain when you try to write a major piece, be it a manuscript or a grant proposal, when you haven't been writing consistently.

I thought this was an appropriate topic as the semester ends, and for many faculty researchers, their daily schedules change for the summer. Certainly, summer is a good time for a little break, but make sure you don't pay for that break with your writing!

Don't wait to be inspired
I'm happy to report that in all of my bad habits, procrastination isn't one. Yet, I must admit that when I have a major writing project on the horizon, I often go through a short spurt of paralysis. I'm not sure how I want to approach the piece, or it feels a bit insurmountable. However, I recognize when I'm doing this and am able to move through it. I do this in one of two ways. If I really am clueless as to my approach, I turn to the literature. You can't write well, if you haven't read the relevant literature and know what you're contributing. The second way is to just start writing. Be careful not to get stuck in the reading where you just keep looking for the next article instead of starting to write.

Do outline and rewrite
As I've mentioned in previous blogs, free-writing can be a good way to get yourself going even if it's really bad in the beginning. However, if you write your first draft completely off the cuff, oftentimes you have a lot of re-working and re-writing to do. Now, you'll always have re-writing to do even if your first draft is awesome. But, if you've written 20 pages without looking back once, who knows what you're working with (including you). By all means, start free writing, but try drafting the introduction of your paper and then working with that to decide what makes sense and what doesn't, what should you keep and what is off topic. Use this initial writing and brainstorming to create an outline for your piece. Look at the nuggets that come out of your initial thinking and writing and decide how you're going to build on those.

Try to write in chunks or sections and make sure you're on track. Having your piece organized in sections makes it easier to re-write and revise your writing. As you finalize your writing, make sure that it flows and those sections fit together. Also, try to avoid correcting your work at a proofing level (grammar, spelling, and punctuation) too early. Stay at a higher level of editing/revising until you think you've got a final draft.

Do consider your writing practice
As you consider your writing practice for the summer, think about what you're doing now. When do you do your best writing? Are you a morning person or a night owl? Try to plan your daily writing around when you're at your best. Also, look at the bigger picture. What is the flow of your week? What days do you have meetings? What days can you spare uninterrupted time to write? Also, be honest about your bad habits. If you procrastinate or get paralyzed on a new project, recognize it and make a plan to get through it. Take all these things into account and make yourself a summer plan. Identify when you'll write and when you'll exercise. :)

How to Develop a Daily Writing Practice - Mattan Griffel
16 Ways to Improve Your Writing Skills  - Dan Shewan

Friday, May 6, 2016

Research collaboration makes it happen

We in ORDE sometimes meet with faculty that have an abundance of great ideas for research for which they are seeking funding. But these great ideas are not always fully developed or in alignment with agency goals. Given the traditional lone-wolf approach in research, many PIs think that if they haven't figured out a whole project by themselves then they won't do it. However, more and more, the most exciting and competitive projects are collaborative. But, if you're not sure how to form these collaborations, here are some tips.

Know your strengths and weaknesses
Before you reach out to collaborators, be clear on where your expertise lies and where it is limited. This will help you to pitch yourself and your project to potential partners and help you to identify the right partner(s).

Identify potential team members' strengths and weaknesses
Once you're clear what your role should be in a project and what roles and needs you must fill to make it come to fruition, you're ready to look for collaborators. Talk to your colleagues and make connections in the areas you have needs. When you identify a potential collaborator, set up an initial conversation to vet them.

Be ready to share
If you want a collaboration to be successful, you shouldn't go into conversations assuming that you're the boss and your collaborator will just provide what's needed. If that's how you want to run things, then you're really looking for a consultant on your project, not a collaborator. Most researchers will not be willing to invest themselves in a project that does not feel like it is theirs. So, be prepared to not only share your idea, but adapt the idea with your new partner(s).

Be ready to assert yourself
On the flip side, a collaboration shouldn't feel like a hand off of your idea to another. Some researchers feel their project is co-opted especially when they seek collaboration with a more seasoned PI. To combat this, be ready to assert yourself and demand a true partnership in the project.

Know what's in it for each researcher
An essential piece to a good collaboration is setting each collaborator up for success. When beginning a collaborative project, it's important to get all motivations out on the table to make sure they're complementary. Also, this is the time to discuss order of authorship of subsequent publications, and who will provide what resources and time to which parts of the project. This can help you avoid confusion or conflict later on.

But is collaboration worth it?
Collaboration is a lot of work, but it seems to be the direction we're going. According to Adams (2012):
  • Co-authored pubs tend to get cited more
  • The first paper with 1,000+ authors was published in 2004
  • The U.S. collaborates with China the most on 3-4% of its Science papers
The Rise of Research Networks - Adams (2012)

Friday, April 29, 2016

The mystery of funding decisions

This week, the National Institute on Aging (NIA) put out a blog detailing their complex use of multiple funding lines to make funding decisions in various priority areas. The NIA does this to offer transparency and clarity to researchers. Basically the funding line identifies the minimum score that will be funded in a particular area, such as Alzheimer's and all grants scored better than the funding line are then funded.

The NIA blog joked that to understand their funding system you "need an advanced degree in mathematics," but that got me thinking about the wide range of ways that different agencies make funding decisions. These decisions are often made using some semblance of the following approaches:

Scores/Peer Review
Many agencies have a rigorous peer review process to assess the science or the approach proposed by the researcher. Some agencies, such as the NIH and NSF, recruit reviewers with the right expertise to assess the proposed science in their grant applications. However, even at these institutions, there are oftentimes grant reviewers who are not experts in your precise area reviewing your grant. This makes it important to strike a balance in your grant writing that speaks to the experts reading your grant, but also is clear and compelling to those who are not experts in your area.

When it comes down to it, not all grants that score well in peer review are funded, and in fact some grants that receive slightly worse scores may get funded ahead of the most stellar if they are a better fit for the agency's priorities. Although PIs sometimes look at agencies as piggy banks, that is certainly not how those in an agency see themselves. Instead of seeing their responsibility as handing out money to do the best research, they see themselves as the stewards of a mission with distinct goals in which they are invested. Thus, it is essential that grant applications incorporate the mission and goals of an agency for them to realize success.

Who you are as the researcher, or who composes your research team needs to be the right fit for the project you're proposing. As we often tell our researchers, you must show reviewers that you are the best person to conduct the research you're proposing. Unfortunately, it's not enough to come up with a truly great research project. You need to have that and you need to have the dream team or be the dream PI to carry it out.

It's also true that in the grants world, for many agencies, it's who you know. Some of our PIs who have been funded by the Department of Defense (DoD) suggest that this is a big part of the DoD's funding decisions. The DoD likes to fund researchers that they know and that they know do good work. This speaks to why it's so important to reach out to and work with Program Officers at agencies when they are available to you. Working with a PO gives you a leg up in understanding what an agency wants, but also lets the PO know you, which can also give you an advantage in some cases.

At many agencies, the Director is the one who gets to make final funding decisions and is charged with making the best decision for the agency. This is true for Program Directors at the NSF and Institute and Center Directors at the NIH. Certainly, this is frustrating to PI's when a Director makes an ultimate decision that does not fund them, but the Director has a purview of all of these other mechanisms and can make the best decision for the agency to further their mission.

Not all agencies use all of these approaches to determine funding, so it's important for you to do your homework on the agency, their mission, goals, and their processes even before you sit down to write your proposal.

Resources (examples of how funding decisions are made):
Transparency and funding lines - NIA Blog
Grant Review Process - National Endowment for the Arts