Bommann, L (Bommann, Lutz); Daniel, HD (Daniel, Hans-Dieter) Convergent validation of peer review decisions using the h index - Extent of and reasons for type I and type II errors JOURNAL OF INFORMETRICS, 1 (3): 204-213 JUL 2007

[Eugene Garfield]

E-mail Address: bommann at gess.ethz.ch; daniel at evaluation.unizh.ch 

Author(s): Bommann, L (Bommann, Lutz); Daniel, HD (Daniel, Hans-Dieter) 

Title: Convergent validation of peer review decisions using the h index - 
Extent of and reasons for type I and type II errors 

Source: JOURNAL OF INFORMETRICS, 1 (3): 204-213 JUL 2007 

Language: English 

Document Type: Article 

Author Keywords: research performance; h index; Hirsch index; convergent 
validity; peer review; type I and type II errors 

Keywords Plus: PREDICTIVE-VALIDITY; COMMITTEE; SELECTION; PUBLICATIONS; 
SCIENTISTS; FAIRNESS; JOURNALS 

Abstract: Hirsch [Hirsch, J. E. (2005). An index to quantify an 
individual's scientific research output. Proceedings of the National 
Academy of Sciences of the United States of America, 102(46), 16569-16572] 
has proposed the h index as a single-number criterion to evaluate the 
scientific output of a researcher. We investigated the convergent validity 
of decisions for awarding long-term fellowships to post-doctoral 
researchers as practiced by the Boehringer Ingelheim Fonds (B.I.F.) by 
using the h index. Our study examined 414 B.I.F. applicants (64 approved 
and 350 rejected) with a total of 1586 papers. The results of our study 
show that the applicants' h indices correlate substantially with standard 
bibliometric indicators. Even though the h indices of approved B.I.F. 
applicants on average (arithmetic mean and median) are higher than those 
of rejected applicants (and with this, fundamentally confirm the validity 
of the funding decisions), the distributions of the h indices show in part 
overlaps that we categorized as type I error (falsely drawn approval) or 
type II error (falsely drawn rejection). Approximately, one-third of the 
decisions to award a fellowship to an applicant show a type I error, and 
about one-third of the decisions not to award a fellowship to an applicant 
show a type II error. Our analyses of possible reasons for these errors 
show that the applicant's field of study but not personal ties between the 
B.I.F. applicant and the B.I.F. can increase or decrease the risks for 
type I and type II errors. (c) 2007 Elsevier Ltd. All rights reserved. 

Addresses: ETH, CH-8092 Zurich, Switzerland; Univ Zurich, Evaluat Off, CH-
8001 Zurich, Switzerland 

Reprint Address: Bommann, L, ETH, Zaehringerstr 24, CH-8092 Zurich, 
Switzerland. 

Cited Reference Count: 44 

Times Cited: 0 

Publisher: ELSEVIER SCIENCE BV 

Publisher Address: PO BOX 211, 1000 AE AMSTERDAM, NETHERLANDS 

ISSN: 1751-1577 

SCIENCE 309 : 1181 2005 

*BOEHR ING FDN
FDN PROGR : 1999 

*NAT AC SCI
QUAL RES SCI : 1982 

*STATCORP
STAT STAT SOFTW REL : 2005 

BORMANN L
J AM SOC INFORM SCI : 1998 

BORNMANN L
Criteria used by a peer review committee for selection of research 
fellows - A boolean probit analysis 
INTERNATIONAL JOURNAL OF SELECTION AND ASSESSMENT 13 : 296 2005 

BORNMANN L
Committee peer review at an international research foundation: predictive 
validity and fairness of selection decisions on post-graduate fellowship 
applications 
RESEARCH EVALUATION 14 : 15 2005 

BORNMANN L
Selecting scientific excellence through committee peer review - A citation 
analysis of publications previously published to approval or rejection of 
post-doctoral research fellowship applicants 
SCIENTOMETRICS 68 : 427 DOI 10.1007/s11192-006-0121-1 2006 

BORNMANN L
Selection of research fellowship recipients by committee peer review. 
Reliability, fairness and predictive validity of Board of Trustees' 
decisions 
SCIENTOMETRICS 63 : 297 DOI 10.1007/s11192-005-0214-2 2005 

CAMPANARIO JM
Peer review for journals as it stands today - Part 2 
SCIENCE COMMUNICATION 19 : 277 1998 

COHEN J
STAT POWER ANAL BEHA : 1988 

COLE JR
WEB KNOWLEDGE FESTSC : 281 2000 

CONROY RM
STATA J 2 : 290 2002 

CRAMER H
MATH METHODS STAT : 1980 

CRONBACH LJ
PSYCHOL TESTS PERSON : 1965 

CRONIN B
Using the h-index to rank influential information scientists 
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY 
57 : 1275 DOI 10.1002/asi.20354 2006 

DAGOSTINO RB
A SUGGESTION FOR USING POWERFUL AND INFORMATIVE TESTS OF NORMALITY 
AMERICAN STATISTICIAN 44 : 316 1990 

DANIEL HD
Publications as a measure of scientific advancement and of scientists' 
productivity 
LEARNED PUBLISHING 18 : 143 2005 

EGGHE L
DYNAMIC H INDEX HIRS : 2006 

EGGHE L
ISSI NEWSLETTER 2 : 8 2006 

EGGHE L
SCIENTIST 20 : 14 2006 

FROHLICH H
BIF FUTURA 19 : 227 2004 

FROHLICH H
BIF FUTURA 16 : 69 2001 

GARFIELD E
LAUNCHING THE ISI ATLAS OF SCIENCE - FOR THE NEW-YEAR, A NEW GENERATION OF 
REVIEWS 
CURRENT CONTENTS 5 : 3 1987 

GLANZEL W
SCI FOCUS 1 : 10 2006 

HIRSCH JE
An index to quantify an individual's scientific research output 
PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF 
AMERICA 102 : 16569 DOI 10.1073/pnas.0507655102 2005 

HOSMER DW
APPL LOGISTIC REGRES : 2000 

KELLY CD
The h index and career assessment by numbers 
TRENDS IN ECOLOGY & EVOLUTION 21 : 167 DOI 10.1016/j.tree.2006.01.005 2006 

KLINE RB
SIGNIFICANCE TESTING : 2004 

LONG JS
REGRESSION MODELS CA : 2003 

MARSHALL E
DOES NIH SHORTCHANGE CLINICIAN 
SCIENCE 265 : 20 1994 

MOED HF
CITATION ANAL RES EV : 2005 

ONG JS
REGRESSION MODELS CA : 1997 

PFEFFER J
EFFECT OF UNCERTAINTY ON USE OF SOCIAL-INFLUENCE IN ORGANIZATIONAL 
DECISION-MAKING
ADMINISTRATIVE SCIENCE QUARTERLY 21 : 227 1976 

ROEDIGER HL
OBSERVER ACAD OBSERV 19 : 2006 

ROEDIGER HL
SCI EXCELLENCE ORIGI : 222 1987 

ROYSTON P
STATA TECHNICAL B RE 1 : 110 1991 

SEGLEN PO
Why the impact factor of journals should not be used for evaluating 
research 
BRITISH MEDICAL JOURNAL 314 : 498 1997 

SPEARMAN C
The proof and measurement of association between two things
AMERICAN JOURNAL OF PSYCHOLOGY 15 : 72 1904 

TAYLOR M
4 INT C PEER REV BIO 2001 

THORNGATE W
MINING ARCHIVES ANAL : 2002 

VANRAAN AFJ
HDB QUANTITATIVE SCI : 19 2004 

VANRAAN AFJ
Comparison of the Hirsch-index with standard bibliometric indicators and 
with peer judgment for 147 chemistry research groups 
SCIENTOMETRICS 67 : 491 DOI 10.1556/Scient.67.2006.3.10 2006 

ZUCKERMAN H
PATTERNS OF EVALUATION IN SCIENCE - INSTITUTIONALISATION, STRUCTURE AND 
FUNCTIONS OF REFEREE SYSTEM
MINERVA 9 : 66 1971