远程服务器返回了意外响应: (400) Bad Request。
NIH firm on grant application rules : Nature News Jump?to?main?contentJump?to?navigationnature.com homepagePublications?A-Z?indexBrowse?by?subjectSearchThis siteAll of nature.com Advanced?searchMy accountE-alert sign upRSS feedSubscribe
Login
nature news homenews archivespecialsopinionfeaturesnews blognature journal Published online 29 March 2011 | Nature 471, 558 (2011) | doi:10.1038/471558a News NIH firm on grant application rules Unpopular policy limiting applicants to two submissions has sped up approvals. Meredith Wadman 
Robert Benezra: “The NIH just ignored the central premise of our argument.”R. Dewitt/MSKCCWhen does no mean no? The answer for researchers applying for funding from the US National Institutes of Health (NIH) is after the second 'no'. Last week, officials at the biomedical agency stood by that policy despite a 20 February petition from 2,335 scientists urging them to restore a system that allowed three tries per project. Some 400 more researchers have since asked to join the campaign.
The NIH's two-strikes-and-you're-out approach to funding grants was implemented in 2009 in an effort to speed up a dilatory review process. Last week, in a letter to petitioners, Sally Rockey, the agency's chief of extramural research, and principal deputy director Larry Tabak wrote that the policy is working as planned. Not only has the average time to approve a grant been reduced by almost half a year, they argue, but the proportion of applicants winning funding on their first attempt has "risen dramatically" (see 'Shorter, faster, better?').
Tabak told Nature that, under the former system, peer reviewers tended to reject meritorious grants that were being submitted for the first time because they knew that the applicants would have two more attempts at funding — in effect setting up a queuing system like that of planes lining up for take-off. The two-strikes policy, he says, "is accomplishing what it set out to do, namely to ensure that we support the strongest science as early as possible".
Click for larger imageRobert Benezra, the cancer biologist at the Memorial Sloan-Kettering Cancer Center in New York who instigated the petition, is not persuaded. After digesting comments from the first 200 petitioners who wrote to him after he circulated the NIH response, he says, "the overwhelming sentiment is frustration with the fact that the NIH just ignored the central premise of our argument".
That premise is that peer reviewers cannot reliably distinguish between an application scoring in the fifth percentile — near the top — and one in the twentieth percentile. Therefore, in a time when limited resources are squeezing the proportion of fundable projects to as low as 5%, reviewers are effectively choosing at random from too many worthy proposals.
"Removing [second resubmissions] has the effect of eliminating outstanding grants that would otherwise be funded in time. In effect, queuing is the only fair way to go when dollars are so scarce," Benezra wrote to the petitioners on Friday. He recommends that the agency define a threshold — he suggests the 25th percentile — and allow grants that score at this level or better a second resubmission.
One of the petition's signatories, Martin Chalfie, a Nobel-prizewinning neurogeneticist at Columbia University in New York City, says that his deepest concern is the effect of the two-strikes policy on young scientists. "There's a learning curve" to grant writing, he says, and "for junior people, we are making that learning curve exceptionally steep".
In their response, Rockey and Tabak counter that the new policy "seems not to have had an effect on new or early-career investigators". In fact, they argue, returning to a three-strikes policy might in fact work against young scientists trying to launch independent labs by lengthening the time it takes to get winning grants through the system.
Although "there is little doubt that some great science is not being funded", Rockey and Tabak wrote, that is because budget pressures have forced the agency to fund an ever-diminishing proportion of applications. "Restoring [second resubmissions] will not change that picture."
ADVERTISEMENT
Despite the groundswell calling for a return to a three-strikes system, some researchers say that even the two-strikes rule does not go far enough. Keith Yamamoto, executive vice-dean of the School of Medicine at the University of California, San Francisco, chaired an advisory committee that recommended that the NIH abolish resubmissions altogether — sparing applicants the time and stress of responding to one set of reviewer criticisms only to be rejected by a new group of reviewers with different opinions. Although the new system is working, says Yamamoto, eliminating resubmissions "would have worked even better".
Tabaka and Rockey note that the agency plans to solicit community opinion on the two-strikes policy in its next survey of extramural grantees late this year.?
CommentsIf you find something abusive or inappropriate or which does not otherwise comply with our Terms or Community Guidelines, please select the relevant 'Report this comment' link. Comments on this thread are vetted after posting. #19408The title could have read "Unpopular policya?|..has sped up approvals by randomly eliminating proposals that were as outstanding as those funded." It was also clear from the petition that we were concerned about young investigators beyond "any explicit grace period". Finally, in an email exchange with Keith Yamamoto recently (witnessed by ~50 others, so I feel justified quoting) he said "To address the problem, we need to do at least two things: first, end the requirement for substantial change, and as I said, I stand firm and confident that the peer review system must not be permitted to determine what investigators are allowed to propose in an R01. My working group did not propose this, and I think it is a misguided and wrong policy. Second, we need to work on institute directors to find any way to increase paylines toward the resolution limit of the system."
I don't think if you can resubmit your proposals verbatim, anyone cares if you call them A1s or A1000s. I can live with this.
Report this comment2011-03-30 08:34:28 AMPosted by: Robert Benezra #19410It seems ludicrous to encourage young investigators (during the grace period), and then yank the carpet out from under them (ending of the grace period) and watch them potentially fail against very large, well funded labs. In tough times like now, a better approach would be to LIMIT investigators to ONE and ONLY ONE R01. This would be much more fair than having some PIs with two, three R01's, which is not really what the R01 was ever intended to achieve. There are alternative funding mechanisms at NIH for larger labs/projects.
Report this comment2011-03-30 08:55:18 AMPosted by: Andor Kiss #19411At a minimum NIH needs to provide detailed information on exactly how they determine whether or not a grant is a 'new' grant. They should provide the exact software they use, the exact limits that they expect in terms of percentages and exactly how this is evaluated to determine whether or not the grant will be considered 'new' by NIH. Investigators should be able to put their 'new' grant through this process themselves, prior to submission, so that they can ensure that the grant will be reviewed, not rejected by CSR, wasting all of the investigator's time and effort during the submission, only to find out months later that the proposal was not reviewed because of this policy. The current descriptions for a 'change in focus and scope' and 'substantially different' are both vague and impossible to discern. Having investigators submit while attempting to adhere to these vague terms only wastes everyone's time.
Report this comment2011-03-30 09:22:23 AMPosted by: Jeffrey Cirillo #19412I regret that the NIH's response failed to acknowledge a scientific fact, which is that distinguishing a 5th %ile from a 15th %ile application is in fact highly random. That is in addition to the important concerns raised about the disproportionate effects on younger investigators.
This means that, while A0 and A1 funding (first submission and first revision) have indeed increased, they have unquestionably done so at the cost of eliminating equally distinguished and deserving projects.
I have benefited from a second revision, and my lab and research program are alive only because of that fact. I hoped for a response which at the very least acknowledged the central points raised in the petition.
Report this comment2011-03-30 09:25:30 AMPosted by: Paul Odgren #19413The NIH has almost entirely ignored the rationale behind our petition, and cited misleading statistics in support of the status quo. Of course initial and 1st resubmission successes will increase if you eliminate 2nd resubmissions, because it's a zero-sum situation. That will also reduce time to support. Eliminate 1st resubmissions and then only initial submissions will be successful, and time to support will be reduced even further. The graph they provided in support of the status quo is insulting.
If they really want to support young investigators, and prevent small labs from being forced to close entirely in times of economic hardship, the NIH should restrict all investigators to one federal grant. The likelihood of that happening is, of course, nil because NIH policies are being guided by those with numerous grants.
Report this comment2011-03-30 09:40:27 AMPosted by: Allen Kennedy #19414As a signatory on the petition, I too was disappointed with the NIH response. As already noted in an earlier post, I would like to add that this problem is compounded by the fact that the NIH does not cap the number of grants that one PI may receive. I recently heard from a fellow faculty member whose collaborator has over 10 current R01s. While this may represent >10 good ideas, I strongly believe that the current system is unfair to the rest of us trying to get one grant, especially new investigators. I'd like to make the point that for the greater good there needs to be a limit on the number of R01 grants that a single lab may receive. Having such a regulation in place would increase the pool of available money to fund new ideas from multiple investigators. Such an approach is badly needed in these austere times.
Report this comment2011-03-30 09:46:38 AMPosted by: Neel Krishna #19415The support to young scientists, as claimed by NIH spokespeople, is actually minimal. First of all, like virgin sex, you are New Investigator only the first time. R29, the FIRST award has been history for years. The closest thing now are R03 and R21; they are called exploratory and such, but in reality, the competition is just as intense, e.g., ex-HHMI investigators are applying for and receiving them; and the expectation of Preliminary Data is just as high, simply because of competition from other applicants who have them. Second, helping the newcomers or offering a better payline for first-time competitive renewals, as many Institutes do, may be well-intended, but is really just lip service unless there is a sustainable support system. Without a long-term funding plan, this is like allowing babies to be born, only to starve them to death later in adulthood. Roughly 3 years down their funded career, these same New Investigators will hate the system as much as we do, when they will run against the A1 wall and drop dead. Perhaps it is also time for a more Socialistic approach. Limit investigators to 2 R01s max, or mandate that they must contribute at least 20% of their effort to make a meaningful contribution to the project. Those who are funded with 2% effort are depriving the same young scientist who, ironically, is running the project for the PI.
We should all do the following. A. Be activists: spend more time on activism such as this, than on lab science. Collectively speaking, we should have started this long ago. Write to our Congressmen and Senators, and write often. Ask them to change our National habit from attacking other countries to attacking diseases. B. Tell our students to NOT choose life science as career. My own children did not. I think, regardless of our ethical or religious beliefs, we will all agree that family planning by contraception is better that death.
The NIH needs to acknowledge that the current system, where ~10% of submitted applications get funded, is simply unsustainable. Reviewers are still prone to overly focus on the Approach, and counting weaknesses biases against innovative or more risky ideas; cookie-cutter science wins. We also need a system that recognizes that the best indicator of future success is past success, and this indicator must be on a per-dollar bases. Currently, reviewers do not have access to Other Support information, and any discussion of an applicant's current support in relation to their productivity is shot down. But shouldn't the productivity of an investigator with 5 RO1s (or equivalent $) be judged differently than someone with 1 RO1? Right now, the system is stacked towards those with lots of funding, as they are more likely to be judged as productive, even if the bang-for-the-buck that they deliver is low.
If we want to encourage innovation and reward productivity, I believe that we should move to a system like HHMI's: lab's would apply for NIH funding, which would be reviewed every 5 years (at which point funding could be increased, decreased or ended after 2 additional years). Finally, another solution to the dwindling NIH budget would be a more equitable distribution of indirect costs: there is simply no good justification for most public universities getting ~50% indirects while well-healed private institutions get ~100% indirects. Cutting all indirects to ~50% would free up a heck of lot of dough for new RO1s.
James DeGregori
University of Colorado School of Medicine.
It is important to recognize that the quality of the review process is limited by the quality and variable interests of the reviewers. I'm sure I'm not alone in turning an unfunded grant application into a funded one despite minimal changes by submitting the application more than once over a several year period. Whereas one set of reviewers did not support the application, the other set found it very strong. I agree with the premise that reviewers are generally unable to discern between the top 5% and 25% of applications. Unless there is an honest, reliable and rigorous mechanism to ensure consistency in the review process, it is unfair to expect young or established investigators to re-work a grant application so that it is "different" by some arbitrary amount. This is an important discussion because careers are on the line, given that many current University administrators view faculty, not as scholars but, simply as a mechanism to acquire indirect costs and salary relief dollars.
Report this comment2011-03-30 10:27:36 AMPosted by: Paul Ratz #19418It seems to me that the central issue here is whether or not study sections can accurately distinguish quality in the top 20% of submissions. If they cannot, this is equivalent to a lottery, and there is no question that in a lottery, you increase your chances of winning by playing repeatedly. If this is the case, perhaps NIH should engage in an actual lottery where the grants in the top 20 percentiles are randomly chosen. Once an application has achieved top 20 status, it would be automatically re-entered until funded. An alternative would be an actual queue, where, having achieved a top 20 status, an application would be guaranteed funding eventually, as soon as it came up in the queue.
On the other hand, if study sections CAN accurately distinguish between applications in this range, a different solution to the problem is required. Since I think we all agree that the top 20% of grants should be funded, and since there is insufficient money now to fund the top 20% of grants, we need to find a way to make the money go farther. This could be done by making grants somewhat smaller, by eliminating some large grant programs that suck money away from investigator-initiated awards, and/or by limiting the total number of grants or total dollars per investigator.
I am concerned that the NIH is treating this issue as if nothing is actually wrong. In fact, we are facing a real crisis. This is a pipeline issue. Young faculty who cannot get grants will lose their jobs. Mid-level and senior investigators who cannot renew grants but already have tenure will retain their jobs but not be able to continue their research. It is unacceptable for these to become routine events, not because the scientists were not good, but because they were not lucky. We should not close our eyes to these issues. We must face them head on and find solutions. Otherwise we face the situation where none of us in good conscience can recommend a career in biomedical science to our graduate students.
Report this comment2011-03-30 10:39:22 AMPosted by: Tom Blumenthal #19419I agree with all the previous points, but I have not seen any discussion of one of the signal strengths of the A2 process, and that is that better science emerges. The current claim of more rapid funding reminds me of the "speed-accuracy" trade-off. Of course, by necessity as pointed out in several previous comments, the new system results in shorter delays to funding, but there is a cost. In my many years of reviewing I saw countless instances of where an application that was not successful as an A0 was as an A2...because it was much better science. Having the benefit of a panel of experts identifying weaknesses in a project, a far superior project emerged. The new system might result in "faster science" but I'll bet it is overall weaker science.
Report this comment2011-03-30 10:49:13 AMPosted by: Marc Branch #19421I too was disappointed with the NIH response, but not surprised. It seems that recent NIH policy changes have made it much more difficult for investigators to obtain what little grant funding remains while at the same time making it much easier for reviewers to pan grants. When NIH changed to the two-strikes policy, the response to the critique was reduced from 3 pages to 1 page. Thus, applicants now have a single Hail Mary Page to respond to all of the bullet-point weaknesses included on the summary statement. The length of the critiques (and the number of weaknesses listed) is not limited. Regarding weaknesses and grant reviews in general today, anything goes. Reviewers can write anything about a grant, right or wrong, scientific or unscientific, and it becomes the PIa??s problem. The review committee is not held to any standard of scientific empiricism and reviews have become increasingly petty and focused on grantsmanship issues. The following comments are from actual summary statements: a??The PI did not take advantage of the richness of the environment at his universitya??, a??The grant does not live up to its titlea??, a??The preliminary data is not sufficienta??, a??The methods of analysis are not innovativea??. What is enough preliminary data? What is innovative? I only mention these issues relating to review because I fear that until the process becomes accountable, it will not matter how many resubmissions we are allowed. If NIH will keep the two-strikes policy in place, then it has the obligation to make the review process scientific and accountable. PIs should also be allowed 3 pages to respond to open-ended critiques of grantsmanship.
Report this comment2011-03-30 11:02:48 AMPosted by: Donald Kuhn #19422A critical issue that was ignored by NIH in their response is what happens to those highly regarded proposals (eg upper 25%tile) that are not funded because of the pay lines these days and which have to be discarded. As far as I can tell, there are no guidelines for what constitutes a "new proposal" and the likelihood that people at the Center for Scientific Review who do not understand the science will be making arbitrary decisions about this vital matter that affects careers of scientists, who are not able to "reinvent" their science without considerable effort and funding. A Catch 22!!
Report this comment2011-03-30 11:03:24 AMPosted by: Bruce McEwen #19426From the NIH response:
virtually all applications that initially scored in the top 20% were eventually funded but only after "waiting" one or two years in the A1/A2 queue
Surely NIH should WANT to fund precisely those grants – the ones that, even on first submission, are clearly in the top 20% !! Their statement seems to me tantamount to an admission that at least some (and probably most) funded A2s were originally in the top 20% & were worth funding. By sunsetting the A2s, those applications are likely to be thrown out.
Although obvious, another huge disadvantage of the new system is that the "dialog" between applicant and reviewers is foreshortened. We get just one shot at convincing a doubtful reviewer. This reduces an already-strained system to a game of bagatelle.
Finally, of course, if we are not permitted to submit the unfunded A1 proposal as a new application to a different review panel, then the IDEAS are thrown out too, consigned forever to the trash heap based solely on a few reviews, which we have had just one chance to rebut.
Report this comment2011-03-30 11:24:34 AMPosted by: J Lindsay Whitton #19427The system is in crisis. There are too many people chasing too few dollars. The best and the brightest students are beginning to realize this and are looking elsewhere for rewarding careers. Given that overall funding is unlikely to increase anytime soon, scarce resources should be used more wisely. Increasing the degree of difficulty ten-fold for each successive R01 per investigator (10%, 1%, 0.1%) would serve two goals-- it would make more room for young scientists, and it would provide a form of birth control for those senior scientists who can't restrain themselves. In general, the more resources one person controls the greater the waste of both money and people. Another area that needs a hard look from the outside is Francis Collins proposal to divert basic research funding to drug development. The pharmaceutical industry depends on NIH-funded basic research that no one else will support. On the other hand lots of small molecule screening efforts at academic institutions have provided scant commercial successes.
Report this comment2011-03-30 11:27:17 AMPosted by: Joe Lipsick #19428As many successful and smart scientists are losing their extramural NIH funding and have or will soon shutter their labs, I see no reason why the intramural program is still functioning as a separate, ridiculously flush research environment with little to no peer review. Let these scientists compete with the rest of the scientific community in the extramural review process. If nothing else, it would be an interesting experiment.
Report this comment2011-03-30 11:28:41 AMPosted by: Jill Schumacher #19429Kudos to Robert Benezra for providing the voice for well over 2,000 investigators, who believe that NIH's two strike policy is not serving the public well. In an effort to hasten work-flow, and in the name of funding the most meritorious proposals, NIH is throwing the baby (the many meritorious proposals that just barely missed the mark) with the bathwater. Can't we all (I mean NIH) just admit that, as those of us who have served on study sections know all too well, reviewing science is not a perfect process?
Report this comment2011-03-30 11:33:29 AMPosted by: Jeffrey Blaustein #19431Thanks to Robert Benezra for being the voice for so many of us. I too was disappointed by the response from the NIH. However, I think we all know the real problem is that there isn't enough funding to support all the meritorious research being submitted. We can change the policies and the reviewing criteria all we want. In the end, lots of meritorious ideas won't be funded.
And while I applaud the emphasis on helping new investigators, it does leave us in quicksand once that first NIH grant is awarded. Mid-stage investigators don't have that extra help anymore, but we aren't established enough to be competitive against well established and funded labs. And without funds, it is hard to me to take my science in new directions when my current projects are rejected. Reviewer requests for even more preliminary data get harder and harder to deal with when there is no money to do the work!
As a community, I think we need to start thinking about more creative ways to spend the money we do have. I think limits on the number of R01s per lab is reasonable. The proposal to ask Universities and Medical schools to pay salaries to their faculty members and allow grants to be used for research also seems like a good idea. Reducing the indirect costs awarded would also allow more money to be given to research. But until we have some solutions to the lack of funds, restricting the ability to apply for funding with a meritorious project doesna??t seem like the wisest course.
Rebecca Burdine
Princeton University
It troubles me greatly that after two negative reviews an investigator's whole line of research can essentially have to shut down, despite the fact that years of serious work likely led him or her to the experiments that were proposed. Those of us who have served on study sections fully understand the imperfect judgments that get made. The determination of whether a line of investigation is worthy, now essentially lies in the hands of 6 people (3 reviewers x2), since it almost always is the assigned reviewers opinions that get followed by the whole study section. Also, how long is the re-submission embargo supposed to last? I've seen no information about that. One year, two years, forever? Advances in a field can happen quickly. An idea not accepted today, or thought of as "too far out" can easily become the gospel tomorrow. We have all seen this happen. The "two strikes and you are out policy" is incredibly repressive and squashes creativity which has been the hallmark of American science. More than ever now people are being forced to propose what they feel the majority will consider acceptable, rather than following what their data may be telling them.
Report this comment2011-03-30 12:07:29 PMPosted by: David Rekosh #19435The biggest problem with the new approach is the requirement that any RO1 submitted after an unsuccessful A1 submission must be substantially different from the A1 version. This makes no sense. As a former Chair and current Dean, my advice to young investigators has always been to establish a well-defined expertise and research theme and stay reasonably focused until you are established in that field. This is not possible with the current system. We are insisting that young investigators NOT stay focused if they get a 10th percentile on their A1 submission. It is equally problematic with senior investigators. I know of multiple cases where an investigator who is a leading authority in a key research area gets an outstanding score on an A1 submission (e.g., 11th percentile) and is then required to shift his/her focus away from experiments that they are uniquely qualified to pursue. I can't imagine that a significant segment of the scientific community thinks these two scenarios represent the best way to drive science forward.
Posted by: John Morrison
OK, I decided to stir the discussion :) . While working for many years in the national DOE laboratory, I was watching the humiliating process of getting NIH grants by my academic colleagues. I could see the adverse effects of the process on their self-esteem, but even worse the changes they did to their initially excellent ideas to please the average picky (often about little details) voting members of the review panel. Now myself in academia I have to go through the same system to get grants. Still my spirit is high (so far). So my opinion is not based on "three strikes and you are out" vs the new two attempts only, but is more general about the whole NIH process, especially in comparison with DOE, NSF etc funding mechanisms. While still at the DOE lab I was invited as an expert in nuclear instrumentation to join some of the "study sections" (first I noticed the funny language used in the process, I would rather call it an "inquisition panel" :) ). After two stints on the panels I quit. And in frustration and even disgust. Why? Because there is no democracy really in expertise. You cannot institute a fair review system where 30 or so panelists of whom only a few are really experts in the matter of the particular proposal can have the same weight ONE vote and then the average score is construed. I noticed that the psychological aspects (yes!) of the review were very important. The silent voting majority is swayed by the relative "oratories" of the key reviewers and vote accordingly. It is simple. They are NOT experts so they have to trust somebody. If there is a difference of opinion between key reviewers, the others will go with a better debater. In one case I was the losing debater to an expert in MRI (nothing against MRI here :) ) who was debating with me the merits of the SPECT (sic!) system proposed, that I, an expert, was for. His negative opinion killed the proposal. I felt like my service effort was a waste. Sorry, but again it looked to me like a politically correct debate where you want to hear from the "other side", even if the other side is ignorant in the subject matter. Like the global warming "debate". Very unscientific.
And then the dreaded "unscore"and throwing (then) the hard copies into the bin of oblivion. And the "lack of enthusiasm" euphemism for "your proposal sucks". Finally the concept of giving more points for worse proposals. Like penalty points.
But seriously, I felt so bad about participation in that unfair, biased, and even just plain stupid process that I quit. Let me assure you that even though the DOE (I got funding from DOE while a professor at UF) and NSF review mechanisms are not perfect, I had much higher comfort level as a reviewer, and grant submitter and receiver, because the system (at least then) involved a small group of real experts, not diluted by a voting consensus-driven silent majority.
Sorry for this opinion from one spoiled ex-government scientist.
The response from the NIH to this petition is frustrating. What they basically tell us is that things are going great, we scientists are just not smart enough to appreciate it. As has already been pointed out, the percentile rankings are quite random within the top 10 to 20 percent and there is no way that a proposal that barely misses the payline (even as an A1) lacks scientific merit and thus does not deserve additional consideration. They completely sidestep this issue! NIH should be on our side and at least acknowledge that there is a problem with some of these recent changes to the application and review process.
Report this comment2011-03-30 12:24:09 PMPosted by: Karl Munger #19442The response from the NIH appears is disappointing, but seems to be consistent with their general approach to the difficult funding situation. In times of limited resources, which may be long term, the NIH will have to decide how best to allocate these restricted funds to maintain the high research capacity and productivity of US scientists. Despite the warning signs for many years (i.e. continued decrease in funding rates and generally flatlined budgets), the NIH has has displayed little flexibility in response to the 'perfect storm' that is now raining on science. Letting funding rates slide below 10% is not only damaging to the scientific capacity of the US, but hinders biomedical economic growth that is spurred by NIH funded laboratory based research. One reasonable change would be for the NIH to consider redirecting funds from other granting mechanisms/projects to support the top 15-20% of RO1 grants, as capping the number of RO1s (although a good idea), will not solve the problem alone. Without the ability of an independent academic research lab to renew or obtain an RO1 that is scored in this top percentile range, the ability of the US to continue to do visionary science and make clinically relevant breakthroughs will be significantly diminished.
Report this comment2011-03-30 12:44:04 PMPosted by: Joshua Corbin #19445It's very unfortunate that NIH is stubbornly sticking with a bad policy. Three additional points that I would make:
1. Applications, and their science, improve as they go from A0 to A1 to A2. It's not just a que, it's selection of the most improved.
2. Many of the newly funded A0's are likely to be reworked applications that missed funding as A1s.
3. The NIH response does not even clarify the criteria for determining what constitutes a new application. If they can't explain the rules to use, how can they expect us to follow them?
Report this comment2011-03-30 12:50:48 PMPosted by: Ed Greenfield #19446What is clear from the NIH response is what we have seen for the past several years – continuing changes to a system that was working without addressing the real problem. First scoring was changed from a 5 to 9 point scale (this does not change the percentage scores – which is all that matters). Next shortening the applications on the theory that they are easier to review (yet this now means young investigators with less experience get hammered on "grantsmanship"). Then changing the review format to bullet points (how can I get a criterion score of 3 from a reviewer with NO weaknesses listed – how can that be addressed?). Reducing the page limit for rebuttal (perhaps rebuttals should be in bullet lists). Now reducing resubmissions.
None of these changes addressed the core issue of reductions in budget and all placed additional burden on PIs and reviewers.
In the past institutes had no issues taking administrative cuts from direct costs. A reasonable approach is:
1. Cut yearly budgets to $200,000 from $250,000
2. Cap indirect cost rates at 55%
3. Require 25% PI effort on R01 or equivalent grants with a 3 grant limit
These 3 changes would lead to a substantial increase in money available to fund investigator initiated grants.
Report this comment2011-03-30 12:56:46 PMPosted by: Derek Schreihofer #19447There is perhaps a larger problem that NIH did not adequately respond to and that arises from the coupling of this limited resubmission policy to the other 'new policy', being that unfunded A1 applications cannot be resubmitted unless the project is completely different. So when an A1 scores in the 15-20 percentile and is not funded, for example, such a meritorious project proposal will not be reviewed and funded by the NIH, forever, as it would appear. That is worst outcome I can imagine. As the petition points out, this places a large burden on new investigators with minimal preliminary data (although all career stages suffer). The combined effect of these policies is the reason that I strongly oppose the current NIH two strikes approach. NIH should admit not only that great science is going unfunded but also that great science is being essentially blacklisted from future funding. I see no reason why A1s receiving a good score (under 25 percentile or such) shouldn't be invited back for an A2.
Report this comment2011-03-30 01:00:13 PMPosted by: Jamey Marth #19448Keith Yamamoto's committee had it right – to do away with resubmissions altogether so that every application is considered on its merits, rather than how well the applicant has accepted reviewers' recommendations or it's status in the queue. There was no suggestion in that report that an applicant be barred from trying again, or that CSR staff should police submissions to decide if they were "new" or "illegal resubmissions".
enhancing-peer-review.nih.gov/.../NIHPeerReviewReportFINALDRAFT.pdf
Since CSR has not been able to define "new", staff are employed to analyze similarity searches and handle appeals while applicants are wasting time redrawing figures and changing words. Drs Tabak (who co-chaired the above report) and Rockey believe that as a study section member my judgment is infallible, but as an applicant I don't know what's important in my field. Let's acknowledge that the truth is somewhere in between and let grants be reviewed by peers, not word-recognition software.
The goal, as I understand it, is to make the peer-review process more fair and more effective to fund the best science. The best way is to keep human beings in the decision process, not arbitary rules and a HAL-9000 computer somewhere in the NIH basement. As a former reviewer, sure, I got tired of having to review and critique the same warmed-over grant 3 or 4 times. But I've also seen grants that struggled as A1's, only to come back as A2's with important breakthroughs that blew the reviewers away and received low-single digit percentiles. Under the new system, those research projects would be delegated forever to the dust bin unless picked up by another funding agency. In my opinion, let the reviewers decide whether an A2 or a new submission, flagged by the NIH computer, should receive a full review after seeing the pink sheet of the prior grant, the new Introduction (if any), and the new Specific Aims page. If the reviewer decides the application is not substantially improved, let them check a box "not substantially improved over earlier application to be competitve". If both reviewers do so, that should be the end of it for that cycle, but with a later breakthrough the P.I. could always submit again.
Report this comment2011-03-30 01:16:38 PMPosted by: Mark Lehrman #19450Your email conversation with Dr. Yamamoto is striking – by ignoring his advice about "substantial change," the NIH has grossly distorted the policy. But his idea about eliminating revisions completely discounts the idea that in many cases, responding to reviewer's comments can strengthen a grant. And by inflating the A0 pool, it would seem to do nothing except eliminate the slight advantage the "last-time-up" A1 and A2 grants used to get.
Report this comment2011-03-30 01:20:07 PMPosted by: Jon Backer #19452FOOD FOR THOUGHT
It is depressing that NIH seems to pay so little attention to the opinions of nearly 3000 of us, who after all are also their peer reviewers. Maybe we should try to get their attention by refusing to serve on study sections, until this ill-thought through policy is changed.
Report this comment2011-03-30 01:57:55 PMPosted by: David Rekosh #19454So many good points. I think some simple things could help. First set minimum percent effort at 25% per grant. No one should be given 10^6 dollars with 5% effort. Second NIH needs to get tough on indirect rates saving more money for the bench top.
Seriously, you can change the rules and protocols all you want NIH but all you are doing is playing a shell game. What needs to happen is increase funding to a level of 25% of all submitted grants. None of us can pick 1% from 5% or even 15% and everyone knows it. We know you have to abide by the budget Congress give you but you NIH get to talk to Congress and justify what you are requesting. Be persuasive, because none of these changes is doing anything to solve the real problems we Scientists face, they merely help randomize further the process well described in this thread.
Report this comment2011-03-30 02:11:59 PMPosted by: Anthony Firulli #19457The response from NIH does not acknowledge or address any of the points in the open letter by Benezra. Nor does it consider any of the positive suggestions made.
As a member of a study section, I know and trust members to make careful decisions. But disagreement among the experts exists; and this is especially true for those proposals that challenge the prevailing views. While the stated goal is to select for research that will be high impact, the funding process tends to work against this very goal.
Furthermore, the fate of a grant hangs on the opinions of 3 individuals. While the reviewers are informed individuals, a sample of 3 is unlikely to provide a statistically valid sample. With a 2nd review only one additional person is added. Thus, the a 3rd submission could improve the sample of opinions. But even with a 3rd review cycle, the review process cannot yield accurate rankings (statistically significant when sampling an adequate population) without being prohibitively expensive. Thus, one must accept that there is some capriousness in the scoring.
Accepting the principle that reviews are imperfect, then one should consider increasing the rigor on a subset of grants that have high merit, but for which scores are statistically indistinguishable. Allowing a 3rd review of proposals that fall in this area where the scores are indistinguishable owing to the disparity of reviews would be one way to address this.
A corollary of imperfections in the quantitative ranking of grants is the likely need to fund more at smaller amounts. This is a difficult area, but the NIH data showed surprisingly modest increases in impact or productivity with increased $. Thus, the optimal size of funding should be scrutinized.
Report this comment2011-03-30 02:38:02 PMPosted by: D. Joseph Jerry #19458I agree with the most of comments posted so far. The only way to achieve some fairness is to remove the limit on re-submission of grants, limit the number of awards given to the individual at one time and make the review process scientific and accountable. Regarding the latter, publications constitute the major if not the only product of science. Should not therefore, the publication record (adjusted for the size of the laboratory and juniority of the scientist) and the scientific citation index serve as a major critical component in the evaluation of the applicantsa?? ability to conduct the research (as opposed to vague criteria of a??noveltya??, a??significancea?? etc.). The troubles with NIH budget are obvious. The present system, forcing scientists to close their laboratories, retire or discard all their previous research accomplishments for the sake of being a??innovativea?? does not solve the problem. It actually magnifies it, because with each previously funded PI going out of business, the money spent by NIH to support this PI turn to a waste. Furthermore, each PI returning to the business after a long period of lack of funding will need a lot more money and time to re-build his/her laboratory, re-hire and re-train the personnell. It is much smarter supporting existing laboratories, than closing and opening laboratories depending on essentially a chance.
Report this comment2011-03-30 02:46:38 PMPosted by: Maxim Dobretsov #19459I read with some interest the comment of Professor Branch with respect to the notion that the sequential revising of grants actually improves the science. I have a nearly diametrically opposite viewpoint. Namely that if we gate the discussion to those applications that eventually win funding in revised form, I found as a reviewer that the "improvement" between the versions was usually in the nature of grantwriting and less often in the nature of fundamental changes to the science. Particularly when one considers that the PI is perfectly free to make the scientific choices as she sees fit after getting funded, it seemed very unlikely to me that much was being accomplished in improving the science. In the case of competing continuation applications, one could compare the previous summary statement to the progress report and come to a similar conclusion about the real effect of reviewer criticisms on what the PI chose to accomplish.
So opinions vary on this topic, quite sharply in this case. The irony, of course, is that Professor Branch and I overlapped terms of service on the exact same study section.
Report this comment2011-03-30 03:05:23 PMPosted by: Michael Taffe #19460It is very interesting that NIH has chosen to disregard the opinion of thousands of NIH-funded scientists. It has been clear for a while now that the new two-tries-and-out system is not working especially under the current pay lines. Additional problems include review panels that have reviewers review too many grants some of which are far-removed from their areas of expertise. The end result is that NIH is indirectly promoting an elitist structure where only a minority of scientists may be funded, not necessarily on the promise of their science, but perhaps increasingly more important on their personal connections and institutional origin. The USA will not remain competitive if only a select number of already well-funded institutions get additional NIH funding. These are important manpower development and equal access issues.
Report this comment2011-03-30 03:06:55 PMPosted by: KONSTANTIN KOUSOULAS #19466I should probably start by saying that I was one of those who did sign the petition a?? not one of these "happy with NIH" people who are closing their eyes on the rotten body of the colossus just because they are "inside the system" and benefit (long time!) from its existence. I should also start by saying that there ARE many decent researchers who are continuously receiving NIH grants and there ARE some decent people working for NIH.
However, the body of this monster is sick with bureaucracy and often (if not always) operates by the well known algorithm: manus manum lavat. Please, if there is anybody here who thinks that this is not true a?? you can throw a stone at me, but I doubt there is a single person with such level of "innocent unawareness". We all know too well what it means when Drs. Rockey and Tabak are saying that "the policy is working as planned." After all, how the "policy" may fail if NIH does not obey The Freedom of Information Act?
Everybody knows that while we can submit our specific complains to NIH, it does not mean that they will be addressed or that NIH will take any actions against its employees a?? in fact, NIH does not have to inform the complaining person about the outcome of investigation (sic!).
What kind of analysis is suggested by Madam Wadman? The "Shorter, faster, better?" graph is ridiculous but it serves the purpose: to dilute our attention and to take it away from the real problem. How many of these "first submissions" (top) are by researchers with established connections with NIH a?? this is the question we must be asking. And a?? what does the "time to approval mean"? Nothing!
I quote: "Tabak told Nature that, under the former system, peer reviewers tended to reject meritorious grants that were being submitted for the first time because they knew that the applicants would have two more attempts at funding a?? in effect setting up a queuing system like that of planes lining up for take-off. The two-strikes policy, he says, "is accomplishing what it set out to do, namely to ensure that we support the strongest science as early as possible". I wonder if Dr. Tabak really thinks that we, researchers are THAT na?ˉve to trust his words. What it means, in reality, is that the "queuing system" is even tougher that it was ever before! It simply means that the "outsiders" will have even less chance to be successful in their competition against those favored by the system.
It looks like Drs. Tabak and Rockey have some kind of a "magic device" to sample and evaluate pretty much everything! I wonder what their confidence is based on when they are saying that the new policy "seems not to have had an effect on new or early-career investigators"?! The Two Moguls are, in fact, counter-attacking by saying that "returning to a three-strikes policy might in fact work against young scientists trying to launch independent labs by lengthening the time it takes to get winning grants through the system." Do they really believe in what they are saying?! There is not an ounce of logic in their words! The Truth, however, is very simple. There are not that many ways of getting into the "system": one has to be a very good scientist AND AT THE SAME TIME be lucky to have his/her proposal reviewed by the panel who UNDERSTANDS the nuts and bolts of the proposal OR to just be "connected".
This brings us to yet another flaw of the system: lack of highly professional, knowledgeable, decent, honest, responsible, professionally-ethical reviewers. It happened way too many times to consider it a "single accident" when high quality proposals were hammered by some ignorant "reviewers", and it did happen too often when the applicant's cry for help and for justice remained unheard. It is all really very simple: those who are NOT SUPPOSED to review cannot decline the offer because they don't want to loose their face or, perhaps, because they really think of themselves as capable to carry the task. At the same time, those who are supposed to select, to control and to provide justicea?| well, they don't really care about anything except the calmness of their environment (otherwise why would they go against FOIA?).
BOTTOM LINE: this is NOT about the number of resubmission, this is about overall health status of NIH as a granting agency a?? and this status is far from being healthy. It is hideously diseased.
And, if you want to learn the truth about NIH, you should read this: "M. Chikindas. 2010. New initiative for humanities granting opportunities: how reality differs from "official information". Journal of Irreproducible Results 51(2): 30-33.":http://foodsci.rutgers.edu/chikindas/MChikindas%20Manuscripts/JIR-2010-51-30.pdf
Report this comment2011-03-30 03:49:37 PMPosted by: Michael Chikindas #19467To determine whether a proposal should be rewritten anew for resubmission should be determined after careful review. Reviewers of a grant in a Study Section (who have arefully reviewed the grant) should be the ones to advise on whether a new rewrite is neccesary. The process now that a new application becomes obligatory after two strikes seems arbitrary and dogmatic. Its not the number. Reveiwers can receommend disapproval of the resubmission of an application even after one round. The key point is reviewer's recommendation should be considered for such a decision. In this way public money can be more effectively used towards funding better science. I do find the response from NIH wanting.
Report this comment2011-03-30 03:59:34 PMPosted by: Robert wong #19470The A2 must be reinstated simply based on the 'noise' in the reviewing system. If it could be guaranteed that the same reviewers would be there on successive study sections, there might be a chance of review in only 2 rounds. However, as you know, this is rarely the case.
Report this comment2011-03-30 04:29:07 PMPosted by: Richard Lieber #19471As a postdoc about to go into the job market, this policy really paints a grim picture of the road ahead. In my view, eliminating the opportunity for a second re-submission does two things that can very seriously affect the launching of a young investigator's career and the development of potentially important scientific ideas. First, as Jon Backer pointed out, critiques from the peer review process are meant to improve the quality of a proposal when appropriately addressed. Limiting this input to a single revision can limit the benefits of having a peer review. If the NIH could at least guarantee that revisions from the first re-submission will be evaluated by the same review panel that wrote the initial critique, then a second resubmission perhaps should be reserved for those scoring in the 10-15 percentile. Second, this two-strikes-and-you're-out policy heavily discriminates in favor of writing craftsmanship (which can be improved by the very experience that this policy restricts), and not necessarily in favor of scientific merit. The latter is practically left to chance, and in many cases to two review panels with conflicting views and not necessarily the most relevant expertise.
Report this comment2011-03-30 04:30:23 PMPosted by: Igor Vivanco #19472I agree with Dr. Robert Benezra that it is practically impossible to distinguish the quality of grants ranking within 20th percentile. This is based on my experience serving on study sections and discussing the issue with numerous colleagues who regularly serve on the study sections. There is also high probability that review process is influenced by personal feelings and connections, zip code effects, professional biases and name recognition. If we go back to the history of NIH funding, there are examples that the grants not considered to be of high priority were turned out to be extraordinary. If a grant is considered to be outstanding, but not funded, why the applicant will not be given another chance and abandon the project. It is easy to impose changes and make sophisticated statistics by the individuals in power at the NIH on their defense, but the reality is that the system is not working and should be taken into consideration very seriously.
Report this comment2011-03-30 04:39:41 PMPosted by: Sudhansu Dey #19477I echo the comments from my colleagues above, especially over how the NIH has not responded to the substance of our letter.
1. It is clear that new investigators have less scope to submit a a??substantially differenta?? application if their A1 submission is not funded, as they typically begin their lab with one main project that they try to get funded. The lack of clear and consistent guidance from the NIH on what a??substantially differenta?? means is a real problem.
2. There is a need for better grant reviewers and/or better training of grant reviewers. I currently serve on a study section and see clear differences in the quality of both written reviews and oral presentations of the proposals during discussion. My own, non-representative anecdotes suggest that the move to bullet point reviews allows poor reviewers to get away with unhelpful reviews. I have seen bullet points in some grant reviews such as a??Data analysis specificitya??, which are completely uninformative and unhelpful, and yet I see no evidence that these reviewers are being asked to change their ways.
3. My personal experience with the people on the ground at the NIH a?? the program officers and the SROs a?? has been overwhelmingly positive. They are doing a tough job in tough times. However it seems that their job, and the job of study section members are being made harder by confusing policy from CSR. For example, our study section has been informed that CSR does not want us to mention the ranking of a grant in our pile of grants (a??this was one of my top two grants this rounda??), or over past review cycles, making it very hard for our fellow reviewers to calibrate our scoring with theirs. Some study sections a??recalibratea?? the grants that are discussed at study section back to the 1-9 range, whereas other study sections do not do this, resulting in most discussed grants occupying a far more narrow range. This is unhelpful to the goal of trying to spread scores and is inconsistent.
4. I agree with my colleagues who complain about the essentially random nature of scoring grants in the top 20%, and I wish the NIH had addressed that specific point. Typically, a study section proceeds in the following way:
- We discuss the new investigator awards.
- We then discuss the small number of grants that everyone loves and get uniformly good scores
- We spend a few hours discussing grants where 2 out of the 3 reviewers are in agreement, leading to the bunching of percentiles between 10-25%. The final percentage of these grants can be heavily influenced by the persuasiveness or advocacy of a single reviewer.
- We then spend even more time discussing grants that have no realistic chance of being funded, but because they have a lower score, there is usually more disagreement between the reviewers. The discussions are longer, more contentious, everyone gets tired and cranky over grants that will not be funded!
As a result of this process, the uniformly excellent grants will be funded no matter what (and so their critiques will be unnecessary), and the rest of the good grants are subject to random scoring variation (and the variability in review quality means that applicants will not necessarily receive a clear signal on how to improve the impact of the grant).
5. Since I have not yet (quite) got the point of proposing lotteries, I would be fine with limiting A2 submissions to grants that score at 25% or better. I would also be fine with ending the requirement for substantial revision, A1s and above and treating every application as new submission.
6. Although I realize this can be a difficult task, I think CSR and its SROs should try even harder to ensure that a re-submission is reviewed by the same reviewers who saw it the first time.
7. I want to thank Robert Benezra for articulating our views and giving up his time to become our spokesman.
Report this comment2011-03-30 05:38:39 PMPosted by: Andrew Groves #19482You did not expect that the bureaucrats who obfuscated grant policy are going to admit they are a failure! Limiting the number of submissions does not benefit anyone since funding is a zero-sum game, and will continue in a downward direction until such time that our increasingly inefficient government thinks science is as worthy of funding as firing 100 cruise missiles at Gadhafi at $1.5M a pop! It does not take rocket science to understand that when funding gets below 15-20 percentile, evaluation becomes a political and random event akin to winning the lottery. This is indeed a sad legacy that will dissuade people from entering into science careers for decades to come. Maybe we need a science tax rather than a gas tax to fund the NIH! Alternatively, the government could cut the salaries of our increasingly inefficient politicians and contribute the money saved to the NIH. Come to think of it, that would only amount to five cruise missiles!
Bob Glazer
Report this comment2011-03-30 07:22:35 PMPosted by: Robert Glazer #19486There is wisdom, but certainly pain, in a 1 or 2-strike NIH grant review system. Investigators will be forced to generate more ideas in different areas for smaller grant proposals. However, researchers may find something better than what they have, or will, on their old, conservative, and beaten mega grants. I think it is a much better style for American science. The low funding rates have been driving investigators and proposals towards the sure bet-conservatism with little innovation. The system should eventually allow the more creative minds in science to prosper. However, an important caution in such a dynamic system is that grant reviewers will need to focus more heavily on an investigator's past track record for discovery, innovation and scientific reliability. The proposals will certainly be more risky to fund, and only the best theoretical and experimental scientists will be successful in this mode. It will also change the lab dynamics. Labs will need a few good 'chiefs'(=thinkers) and far fewer 'indians'(=hands).
Report this comment2011-03-30 08:26:34 PMPosted by: Paul Gardner #19495There are many excellent points made in these comments. The issue regarding A2s could be simply alleviated by relaxing the criterion of a??significant change.a?? But this will not address the more fundamental underlying problem that there are too few funds available to support the number of meritorious R01 applications. The proportion of the NIH budget devoted to R01s has steadily declined over the past decade. Unless and until the economy improves, we cannot expect to see a substantially bigger NIH budget, so the real debate should be about priorities. It frustrates me to see many high quality R01 proposals rejected while billions are spent on centers and the latest technological fads. Translational research may be an easier sell to the public (and to Congress) than basic research, but basic research provides the raw material for scientific progress. The scientific community desperately needs the NIH to reassess its priorities.
Report this comment2011-03-31 01:12:25 AMPosted by: Ferric Fang #19500I think the ends justifies the means here. It's clear the NIH is understaffed and overwhelmed by submissions, so the idea is solely to encourage a higher standard of care in the submissions. Being part of the funding side of things, it's extremely hard to decide who gets what, and each case has to be judged on the submitted application. The funding arm is the last one to say cute things or make everyone smile, but just like scientists, we're overworked and underfunded.
Report this comment2011-03-31 04:26:49 AMPosted by: Ben Collins #19517Frankly speaking, I cannot understand how so many of my fellow scientists fail to see the forest for the trees! In fact, I applaud the monstrosious bureaucrats from NIH for their ability to fool so many people with (I presume) such a high "collective IQ"!
PLEASE, HEAR ME: I repeat that ultimately this is NOT about the number of allowed resubmissions! This is NOT about NIH being "understaffed and overwhelmed". This is NOT about NIH being "overworked and underfunded" - and I shall not pity these government-paid bureaucrats for that. And if it is "extremely hard to decide who gets what" - it means that the people who cannot be professional MUST step down from their positions.
At large, NIH as a granting agency fails to act as an effective, skilled, ethical, professional and unbiased entity.
What does Ben Collins (above) mean when saying that "the ends justifies the means here"? Good things should never be achieved by taking wrong or bad steps! I wonder if Ben Collins will justify manslaughter as a mean to control overpopulation.
And, while there are some decent people working at NIH and with NIH, the system at large is rotten. In an ideal world, following actions would be considered as appropriate:
1. remove incompetent bureaucrats;2. conduct thorough investigation and remove those who were acting as a "good boys club";3. replace the removed employees with carefully selected individuals, based on their specific qualities such as professional and management skills, ethics, etc.;4. develop mechanism for attracting highest quality reviewers;5. make NIH to obey the Freedom of Information Act.And of course, in the real world, all of the above is simply impossible because we all know too well who we are dealing with here.
Report this comment2011-03-31 09:44:01 AMPosted by: Michael Chikindas #19519"READ THE WHOLE TRUTH ABOUT NIH HERE":http://foodsci.rutgers.edu/chikindas/MChikindas%20Manuscripts/JIR-2010-51-30.pdf/url/to/link/to
Report this comment2011-03-31 09:50:37 AMPosted by: Michael Chikindas #19523I was one of the investigators who signed the petition. I have to say that I was very disappointed by the very superficial and poorly argued response that we received from the NIH. Clearly, there are major issues with the policy, which eliminated A2 revisions a?? our argument was presented at the petition and I dona??t want to repeat it here. However, elimination of A2 revisions is not the only problem with the new a??enhanceda?? peer review system.
The new peer review system clearly has some good features. These include 9-point scale, scoring of individual criteria, structured applications and comments, as well as funding of new investigators on the same level as senior investigators. However, the new system also has a number of serious flaws, which make the entire review process flawed.
1) Shorter proposal format essentially eliminated the background section. Some background information still can be added in the Approach section, but the space is so limited that only a very small amount of highly pertinent information can be squeezed in. The idea behind this policy was that an expert reviewer must have sufficient knowledge of the field, and s/he would not need to read a literature review to conduct a professional evaluation of a proposal. This would make sense if the reviewers were indeed experts, but in most cases they are not. This would also work if the reviewers were willing to spend time and learn about the topic of the proposal, but in most cases they dona??t. The reality also is that the increasing number investigators work in the areas, which are not covered by regular study sections, but their grants are reviewed by the regular study sections anyway a?? ad hoc reviewers are rarely invited. This results in a very superficial unprofessional and biased review process. This would not be, perhaps, such a big deal if the number of resubmissions would not be limited to just one a?? one would hope that somewhere along the way the proposal would be reviewed by the experts and receive a fair evaluation; however, the elimination of A2s made that very unlikely. My recent personal experience with the new peer review system was horrific. My R01 went through two rounds of peer review by two different study sections in 2010. I was absolutely stunned by how ignorant reviewers were. I could get more professional comments from people on the street. If the background section was still there, it could be used to educate these a??reviewersa?? about the field and, perhaps, spare me the burden and frustration of reading ignorant comments. But the real issue here is not the background section or elimination of A2 revisions. The real issue is inability or unwillingness of the CSR staff to do their job, which is to ensure expert and fair review of NIH proposals. Just last week, I had the pleasure of finally reaching my SRA after several e-mails and voice mails (and several weeks waiting for him to call me back) just to hear from him that he cannot invite expert reviewers for every grant, which is reviewed by that study section. But what about expert and fair review proclaimed in the CSRa??s mission statement?!
2) Scores for individual criteria seemed like a good idea, but what is the point in assigning scores for those criteria if the final impact score has absolutely nothing to do with these individual scores?! It appears that the NIH is not quite consistent here. If you introduced individual scores, why dona??t you calculate the impact score as the average of individual scores? Apparently, the NIH staff recently realized that they have a problem with the impact score and introduced the summary paragraph, but this had absolutely no effect on how the impact score is assigned.
3) Bulleted comments also seemed like a good idea, but in reality it boiled down to often meaningless and dry statements, which have little value for the applicant.
My overall impression is that the new a??enhanceda?? peer review system was engineered by the senior investigators, administrators and reviewers to make THEIR life easier and reduce administrative burden. Shorter applications and bulleted comments were clearly introduced at the request of senior investigators/reviewers, and had extremely negative impact on the quality of the review process. The shorter application format and non-expert reviewers do not match. It made the review process more unpredictable, less professional and more biased. For me as a new investigator, who received an R21 grant under the old system and had recent experience with the new system, the old system was overall much better than the new one primarily because of the background section, detailed comments and A2s – the A2s were especially important for new investigators who are usually not very experienced in writing grants. The old system favored quality over speed, while the new system favors speed over quality. The irresponsible and unprofessional CSR staff has been a real disappointment. If the NIH really wants to improve the quality of the peer review system and make it fairer, they should pay more attention to the quality of reviewers, reorganize study sections to more accurately reflect current fields of study, reintroduce the background section a?? lengthen the application, rethink how the impact score is calculated and make it transparent, and finally return the A2 revision at least for the new investigators. It is nice that the NIH will address these issues in its next survey of extramural grantees late this year. Hopefully, there will be enough of us left to conduct the survey.
Report this comment2011-03-31 11:57:03 AMPosted by: Andrei Tkatchenko #19525First, I want to applaud Dr. Benezra for initiating this discussion and other participants for their suggestions. I really like cutting indirects and limiting the number of RO1s, but I am afraid this will not be enough. Current science is too specialized for the NIH review system, and in many cases my reviewers are working in the same field and even in the same area (how else can the proposal be reviewed?), while other members of the study section have only a vague idea about the substance, let alone innovation, etc. As the result, the esteemed study section becomes a pliable tool in the hands of two or three main reviewers, creating a typical Potemkin village of a collegial decision. The paucity of experts is also fraught with COI, which is so difficult to formalize. If we define the task as selecting the best projects in a progressively more specialized reaesrch area, maybe the experience of European institutions should be followed? Probably most of us received requests from funding bodies in Europe to evaluate the potential of success of a specific project. This approach allows elimination of COI
I kindly but firmly disagree with vic lev on following:
We cannot cut indirect costs because this is the only thing which makes our universities (read: their administration) "loving" us. After all, it is all about business and profit.R01 proposals have to have a right for existence – otherwise how can one finance a large scale investigation.If "current science" is "too specialized" for the NIH review system, it means that the review system is inappropriate and must be changed.As for the study sections, except for some rare occasions, they are just the toys in the hands of the NIH decision makers. And to make their decisions, these people don't really need that many of truly educated and professionally ethical panel members.
I have some experience in reviewing proposals for NIH, NSF, USDA, BARD, and various science foundations in Europe, South Africa, Ireland etc. In my opinion, European system is much better and carries much less flaws than what NIH offers.
Of course, it would be childish of me to dream for NIH changing their habits, firing bureaucrats and incompetent people, learning from the positive examples across the pond and becoming an unbiased agency. Still, no one can forbid me to dream.
http://foodsci.rutgers.edu/chikindas/MChikindas%20Manuscripts/JIR-2010-51-30.pdf
Report this comment2011-03-31 04:16:49 PMPosted by: Michael Chikindas #19534Many thanks to Dr.Robert Benezra for initiating this very healthy discussion. In addition to the serious concerns regarding the elimination of A2s for the R grants, it is highly disappointing that the rule now also applies to the F31-F32 pre- and postdoctoral fellowships. This obviously sends a most demoralizing message to the youngest scientists at a moment when this country is already losing leadership in scientific education and research.
Report this comment2011-03-31 06:37:15 PMPosted by: carmen sato-bigbee #19540All this focuses on the narrow aspects of the petition (which I signed) or even the overhaul of peer review, i.e., what one SRO termed "rearranging the deck chairs as the Titanic tilts in the water".
The overhaul has many good effects – 70% good, but some 30 % really bad, such as the elimination of the -A2 and erratic, crazy denials of 'new' status to some submissions.
One big problem in all this is the reflexive refusal to bend coupled to tossing out what often seem like smoke screens in NIH responses, leading to an erosion of confidence in how they generate and use analyses to guide decision-making.
The most marked changes they advertise in their response letter are for FY2009 and FY2010. What is not said is if these numbers include R01's awarded under ARRA – after which a lot of lemmings are going to get shoved off "The Cliff".
Ditto in terms of the "Time to Award" in which the NIH response takes pride. Moreover, the NIH data of their response's Fig. 1 & 2 are essentially self-fulfilling prophecies to the extent there are mandates to fund to set ratios of "new" vs "old" (more on that in a moment) investigators. Stated conclusions ("thus allowing them to conduct independent research earlier in their careers" are not supported by the data they show [the pertinent data would be what has happened to median age at first R01, excluding all ARRA-funded R01's].
In the bigger picture, the really tough issues seem to me include (not necessarily in order of priority or unpopularity):
1. If one looks at the stated rationales and written record about the whole 'preferential funding of ESI / new investigators', what it appears to represent is the NIH (a branch of our Federal government) engaging in age discrimination as far as awards of big chunks of $$$. Maybe it's legal for the Feds to discriminate in ways that are illegal for everyone else, but it is disappointing.
2. The NIH response to the petition (and other written records) take an odd pride in pointing to an ever-increasing rate of adding to the pool of R-funded investigators, which is totally unsustainable and at a managerial level seems to be madness other than if the model is "the disposable team leader". In FY 2008, NIH managed to fund a bit over 26,600 R01-equivalent grants (down from 28,743 in FY2003). If one adds 1600 new investigators per year to the pool (NIH's FY2010 number), well, y'all can do the math. I would never argue for sinecures, especially not in the award of funding (though some big labs almost seem in that state for RFA responses, etc). But even at a 40% 'drop-out' rate, as the number of grants continues to decline their artificially boosted production of new R01 awardees is not sustainable even with NIH's apparent desire to squeeze out most current PI's.
3. Since no one expert in the process has much faith in the resolution power between an 8th percentile and, say, a 12th or 16th percentile rating, the rational and 'strong managerial' approach is to select awards of similar 'programmatic value' (a good place for some networking, but . . .) and percentiles in this range based on total cost [direct + F&A (aka 'indirect'].
Are the taxpayers well served by funding something non-modular (let's say, $300,000 / yr direct and a 120% F&A rate) at 7.9 percentile rather than science of programmatic value that is at 10.1 percentile and needing only 10 modules + a 55% F&A rate?
This is a public forum. Please keep to our Community Guidelines. You can be controversial, but please don't get personal or offensive and do keep it brief. Remember our threads are for feedback and discussion - not for publishing papers, press releases or advertisements.
You need to be registered with Nature to leave a comment. Please log in or register as a new user. You will be re-directed back to this page.
Log in / register comments on this story Stories by subjectLab lifePolicy Stories by keywordsNational Institutes of HealthResearch project grantsFundingSuccess ratesPeer review This article elsewhere Blogs linking to this articleAdd to ConnoteaAdd to DiggAdd to FacebookAdd to NewsvineAdd to Del.icio.usAdd to Twitter Recent activitymost recent stories Crashing comets make rings ripple31 March 2011Muscular dystrophy findings fuel French stem cell debate
31 March 2011New Drugs for Hepatitis C on the Horizon
31 March 2011Dying for a long life
30 March 2011Seven days: 25–31 March 2011
30 March 2011 commented storiesGive postdocs a career, not empty promises 119 comments02 March 2011NIH firm on grant application rules 52 comments29 March 2011Virology: Fighting for a cause 24 comments14 March 2011Concerns over nuclear energy are legitimate 20 comments30 March 2011How the penis lost its spikes 14 comments09 March 2011 Related stories Shorter NIH grant form launches06 January 2010 Wellcome Trust makes it personal in funding revamp11 November 2009 NIH responds to critics on peer review11 June 2008 nature jobsScientist in Neutron SpectroscopyPaul Scherrer InstitutVilligen, SwitzerlandPostDoc PositionMax-Planck-Institut fur Kognitions- und Neurowissenschaften LeipzigLeipzig, GermanyMore science jobsPost a job for freenature events 2011 International Social Insect Genomics Research Conference 30 June 2011 — 02 July 2011 Yankui Road, Shenzhen, Guangdong, 518083, China More science events Resources PDF FormatSend to a FriendReprints & PermissionsRSS Feeds external links Robert Benezra NIH grant resubmission policy beginning 25 January 2009 NIH Enhancing Peer Review TopNatureISSN: 0028-0836EISSN: 1476-4687About NPGContact NPGRSS web feedsHelpPrivacy policyLegal noticeAccessibility statementTermsNature NewsNature jobsNature AsiaNature EducationAbout Nature NewsNature News SitemapSearch:? 2011 Nature Publishing Group, a division of Macmillan Publishers Limited. All Rights Reserved.partner of AGORA, HINARI, OARE, INASP, CrossRef and COUNTER
没有评论:
发表评论