New paper on peer review

[Bornmann Lutz]

Dear colleagues:

You might be interested in our new paper:

Bornmann L, Wallon G, Ledin A (2008) Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes. PLoS ONE 3(10): e3480. doi:10.1371/journal.pone.0003480

You can download this Open Access paper from: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0003480

Abstract: Does peer review fulfill its declared objective of identifying the best science and the best scientists? In order to answer this question we analyzed the Long-Term Fellowship and the Young Investigator programmes of the European Molecular Biology Organization. Both programmes aim to identify and support the best post doctoral fellows and young group leaders in the life sciences. We checked the association between the selection decisions and the scientific performance of the applicants. Our study involved publication and citation data for 668 applicants to the Long-Term Fellowship programme from the year 1998 (130 approved, 538 rejected) and 297 applicants to the Young Investigator programme (39 approved and 258 rejected applicants) from the years 2001 and 2002. If quantity and impact of research publications are used as a criterion for scientific achievement, the results of (zero-truncated) negative binomial models show that the peer review process indeed selects scientists who perform on a higher level than the rejected ones subsequent to application. We determined the extent of errors due to over-estimation (type I errors) and under-estimation (type 2 errors) of future scientific performance. Our statistical analyses point out that between 26% and 48% of the decisions made to award or reject an application show one of both error types. Even though for a part of the applicants, the selection committee did not correctly estimate the applicant's future performance, the results show a statistically significant association between selection decisions and the applicants' scientific achievements, if quantity and impact of research publications are used as a criterion for scientific achievement.

Regards,
Lutz

-------------------------------------------------------------------------------------------------------------
Dr. Lutz Bornmann
ETH Zurich, D-GESS
Professorship for Social Psychology and Research on Higher Education
Zaehringerstr. 24 / ZAE
CH-8092 Zurich
Phone: 0041 44 632 48 25
Fax: 0041 44 632 12 83
Skype: lutz.bornmann
http://www.psh.ethz.ch/ 
bornmann at gess.ethz.ch
 
Download of publications: www.lutz-bornmann.de/Publications.htm