Past performance, peer review, and project selection: A case study in the social and behavioral sciences: PDF version

[James Hartley]

Re - David's point that research proposals should be judged on their merit, more or less independently of past performance.

Some (most?) research councils in the UK require applicants to list their previous applications to them, and their success or not at obtaining a grant from them - so it is hard not to imagine that previous performance plays a part in the refereeing process - undesirable as this might be!

Jim 

James Hartley
School of Psychology
Keele University
Staffordshire
ST5 5BG
UK
j.hartley at psy.keele.ac.uk
http://www.keele.ac.uk/depts/ps/people/JHartley/index.htm
  ----- Original Message ----- 
  From: David Wojick 
  To: SIGMETRICS at listserv.utk.edu 
  Sent: Sunday, April 05, 2009 10:26 PM
  Subject: Re: [SIGMETRICS] Past performance, peer review, and project selection: A case study in the social and behavioral sciences: PDF version


  Adminstrative info for SIGMETRICS (for example unsubscribe): http://web.utk.edu/~gwhitney/sigmetrics.html 
  Dear Peter van den Besselaar,

  This is very interesting, but after reading only the abstract I find it puzzling. There seems to be an assumption that past performance should be a significant factor in the success of proposals. One hopes that proposals are selected on their own merit, more or less independently of past performance. If so then the empirical question is whether past performance influences the quality of present proposals, not whether it influences their selection. 

  Moreover, one should not be surprised to find that past performance does not correlate with present quality, for a variety of reasons. For example, past performance may be based on important discoveries that only occur once or a few times for a given researcher, so new proposals weaken with time. Or the focus of science may shift so that past discoveries, and their performers, are no longer relevant. In other words, the dynamics of science might tend to work against a correlation between past performance and proposal selection. If so then the lack of such correlation is not a criticism of the proposal selection body. Past performance and present quality of proposals are simply independent variables. But perhaps I misunderstand the abstract.

  Cheers, David

  David Wojick, Ph.D.
  391 Flickertail Lane 
  Star Tannery VA USA
  http://www.osti.gov/innovation/

  Apr 5, 2009 04:23:27 PM, SIGMETRICS at listserv.utk.edu wrote:


    Adminstrative info for SIGMETRICS (for example unsubscribe):
    http://web.utk.edu/~gwhitney/sigmetrics.html

    Past performance, peer review, and project selection: A case study in the social and behavioral sciences 

    Peter van den Besselaar & Loet Leydesdorff

    Abstract
    Does past performance influence success in grant applications? In this study we test whether the grant allocation decisions of the Netherlands Research Council for the Economic and Social Sciences correlate with the past performances of the applicants in terms of publications and citations, and with the results of the peer review process organized by the Council. We show that the Council is successful in distinguishing grant applicants with above-average performance from those with below-average performance, but within the former group no correlation could be found between past performance and receiving a grant. When comparing the best performing researchers who were denied funding with the group of researchers who received it, the rejected researchers significantly outperformed the funded ones. Furthermore, the best rejected proposals score on average as high on the outcomes of the peer review process as the accepted proposals. Finally, we found that the Council under study!
     successfully corrected for gender effects during the selection process. We explain why these findings may be more general than for this case only. However, if research councils are not able to select the ‘best’ researchers, perhaps they should reconsider their mission. In a final section with policy implications, we discuss the role of research councils at the level of the science system in terms of variation, innovation, and quality control. 

    PDF version available at:
    http://home.medewerker.uva.nl/p.a.a.vandenbesselaar/bestanden/20090327%20magw.pdf


    Peter van den Besselaar
    ---------------------------------------
    professor, head of department

    Address: 
    Rathenau Instituut
    Dpt. Science System Assessment
    PO Box 95366, 2509 CJ Den Haag, The Netherlands

    email: p.vandenbesselaar at rathenau.nl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20090406/1874bed4/attachment.html>