Past performance, peer review, and project selection: A case study in the social and behavioral sciences: PDF version

[Ronald Rousseau]

I do not agree with David. Past performance (but certainly not  
restricted to grant applications) is proof of what one is able to do.

A research proposal - taken on its own - only shows if one has the  
skills to make credible promises.

Leaving young scientists aside  (their case is about giving chances to  
prove what they are capable of) I'd always go for proven merit (in  
recent years).

Ronald Rousseau





>
> Re - David's point that research proposals should be judged on their  
>  merit, more or less independently of past performance.
>
> Some (most?) research councils in the UK require applicants to list   
> their previous applications to them, and their success or not at   
> obtaining a grant from them - so it is hard not to imagine that   
> previous performance plays a part in the refereeing process -   
> undesirable as this might be!
>
> Jim
>
> James Hartley
> School of Psychology
> Keele University
> Staffordshire
> ST5 5BG
> UK
> j.hartley at psy.keele.ac.uk
> http://www.keele.ac.uk/depts/ps/people/JHartley/index.htm
>   ----- Original Message -----
>   From: David Wojick
>   To: SIGMETRICS at listserv.utk.edu
>   Sent: Sunday, April 05, 2009 10:26 PM
>   Subject: Re: [SIGMETRICS] Past performance, peer review, and   
> project selection: A case study in the social and behavioral   
> sciences: PDF version
>
>
>   Adminstrative info for SIGMETRICS (for example unsubscribe):   
> http://web.utk.edu/~gwhitney/sigmetrics.html
>   Dear Peter van den Besselaar,
>
>   This is very interesting, but after reading only the abstract I   
> find it puzzling. There seems to be an assumption that past   
> performance should be a significant factor in the success of   
> proposals. One hopes that proposals are selected on their own merit,  
>  more or less independently of past performance. If so then the   
> empirical question is whether past performance influences the   
> quality of present proposals, not whether it influences their   
> selection.
>
>   Moreover, one should not be surprised to find that past   
> performance does not correlate with present quality, for a variety   
> of reasons. For example, past performance may be based on important   
> discoveries that only occur once or a few times for a given   
> researcher, so new proposals weaken with time. Or the focus of   
> science may shift so that past discoveries, and their performers,   
> are no longer relevant. In other words, the dynamics of science   
> might tend to work against a correlation between past performance   
> and proposal selection. If so then the lack of such correlation is   
> not a criticism of the proposal selection body. Past performance and  
>  present quality of proposals are simply independent variables. But   
> perhaps I misunderstand the abstract.
>
>   Cheers, David
>
>   David Wojick, Ph.D.
>   391 Flickertail Lane
>   Star Tannery VA USA
>   http://www.osti.gov/innovation/
>
>   Apr 5, 2009 04:23:27 PM, SIGMETRICS at listserv.utk.edu wrote:
>
>
>     Adminstrative info for SIGMETRICS (for example unsubscribe):
>     http://web.utk.edu/~gwhitney/sigmetrics.html
>
>     Past performance, peer review, and project selection: A case   
> study in the social and behavioral sciences
>
>     Peter van den Besselaar & Loet Leydesdorff
>
>     Abstract
>     Does past performance influence success in grant applications?   
> In this study we test whether the grant allocation decisions of the   
> Netherlands Research Council for the Economic and Social Sciences   
> correlate with the past performances of the applicants in terms of   
> publications and citations, and with the results of the peer review   
> process organized by the Council. We show that the Council is   
> successful in distinguishing grant applicants with above-average   
> performance from those with below-average performance, but within   
> the former group no correlation could be found between past   
> performance and receiving a grant. When comparing the best   
> performing researchers who were denied funding with the group of   
> researchers who received it, the rejected researchers significantly   
> outperformed the funded ones. Furthermore, the best rejected   
> proposals score on average as high on the outcomes of the peer   
> review process as the accepted proposals. Finally, we found that the  
>  Council under study!
>      successfully corrected for gender effects during the selection   
> process. We explain why these findings may be more general than for   
> this case only. However, if research councils are not able to select  
>  the ?best? researchers, perhaps they should reconsider their   
> mission. In a final section with policy implications, we discuss the  
>  role of research councils at the level of the science system in   
> terms of variation, innovation, and quality control.
>
>     PDF version available at:
>       
> http://home.medewerker.uva.nl/p.a.a.vandenbesselaar/bestanden/20090327%20magw.pdf
>
>
>     Peter van den Besselaar
>     ---------------------------------------
>     professor, head of department
>
>     Address:
>     Rathenau Instituut
>     Dpt. Science System Assessment
>     PO Box 95366, 2509 CJ Den Haag, The Netherlands
>
>     email: p.vandenbesselaar at rathenau.nl
>



-- 
Ronald Rousseau
President of the ISSI
KHBO - Association K.U.Leuven
Industrial Sciences and Technology
Zeedijk 101 -  8400  Oostende,  Belgium
Professor associated to K.U.Leuven
Honorary Professor Henan Normal University (Xinxiang, China)
Adjunct professor of Shanghai University
Guest Professor at the National Library of Sciences CAS (Beijing)
Guest Professor at Dalian University of Technology
E-mail: ronald.rousseau at khbo.be
web page:  http://users.telenet.be/ronald.rousseau

In scientific affairs one can never be too generous.



----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.