Epstein RJ "Journal Impact Factors do not equitably reflect academic staff performance in different medical subspecialties" Journal of Intestigative Medicine 52(8):531-536 December 2004.

[Eugene Garfield]

Richard J. Epstein : E-mail Addresses: repstein at hku.hk

Title: Journal impact factors do not equitably reflect academic staff
performance in different medical subspecialties

Author(s): Epstein RJ

Source: JOURNAL OF INVESTIGATIVE MEDICINE 52 (8): 531-536 DEC 2004

Cited References: 26

Abstract:
Background: The simplest variables to quantify on an academic curriculum
vitae are the impact factors (IFs) of journals in which articles have been
published. As a result, these measures are increasingly used as part of
academic staff assessment. The present study tests the hypotheses that IFs
exhibit patterns that are consistent between journals of different
specialties and that these IFs reflect the quality of staff academic
performance.

Methods: The IFs of a sample of journals from each of four medical
specialties-medicine, oncology, genetics, and public and occupational
health-were downloaded from the Science Citation Index and compared. Overall
and specialty-specific journal IF frequencies were analyzed with respect to
distribution patterns, averages, and skew.

Results: Approximately 91% of journal IFs fell within the 0 to 5 range, with
97% being less than 10. The overall IF distribution featured a positive skew
and a mean of 2.5. Separate analysis of the journal specialty subsets
revealed significant differences in IF means (genetics 3.4 > oncology 3.1 >
medicine 2.0 > public health 1.6; p <.006), all of which well exceeded the
respective IF medians. Journals from the general medicine category exhibited
both the lowest IF median (0.7) and the most positively skewed distribution.

Conclusion: The distribution of IFs exhibits degrees of skew, numeric
average, and spread that differ significantly between journal specialty
subsets. This suggests that factors other than random variations underlie
much of the IF variation between specialty journals and reduces the
plausibility of a reliable correlation between IFs and the quality of
academic staff performance. It is concluded that a dominant emphasis on IFs
in academic recruitment and promotion may select for long-term faculty
characteristics other than academic quality alone.

EXCERPT FROM PAPER :
These conclusions strengthen and extend the concerns raised by many others
concerning the hazards of overreliance on impact factors as academic quality
surrogates.  In an analysis of 204 published articles, Callaham and
colleagues found that traditional "quality" measures of study methodology
and design were poorly predictive of both citation frequency (i.e. impact)
and the IF of the publishing journal;  In fact, the only variable to prove
highly predictive of article citation frequency was the IF of the publishing
journal, suggesting that chance or nonquality variables relating to
manuscript acceptance (eg. newsworthiness as perceived by the journal or the
persistence or "gamesmanship" of an author) may determine the ultimate
impact of a submission.  In addition, Rostami-Hodjegan and Tucker
demonstrated highly skewed citation frequency distributions of individual
articles within journals of a given IF, casting further doubt on the
interpretability of IFs as a measure of individual research output.  Healy
and Cattell reported that industry-linked studies on a given subject tend to
have a sixfold higher IF than non-industry-linked studies, raising potential
concerns about the long-term objectivity and interpretability of the present
system.


Addresses: Epstein RJ (reprint author), Univ Hong Kong, Dept Med, Queen Mary
Hosp, Professorial Block,Pokfulam Rd, Hong Kong, Hong Kong Peoples R China
Univ Hong Kong, Dept Med, Queen Mary Hosp, Hong Kong, Hong Kong Peoples R China

E-mail Addresses: repstein at hku.hk

Publisher: B C DECKER INC, 20 HUGHSON ST SOUTH, PO BOX 620, L C D 1,
HAMILTON, ONTARIO L8N 3K7, CANADA
Subject Category: MEDICINE, GENERAL & INTERNAL; MEDICINE, RESEARCH &
EXPERIMENTAL
IDS Number: 887TD
ISSN: 1081-5589

ANGELL M, 1986, ANN INTERN MED, V104, P261
   ARCHER ZA, 2003, OBES RES, V11, P1376
   CALLAHAM M, 2002, JAMA-J AM MED ASSOC, V287, P2847
   CLARK J, 2004, BRIT MED J, V328, P604
   FEINBERG AP, 1983, ANAL BIOCHEM, V132, P6
   GLADWELL M, 2003, TIPPING POINT
   GOLDBECKWOOD S, 2000, BRIT MED J, V320, P591
   HEALY D, 2003, BRIT J PSYCHIAT, V183, P22
   HECHT F, 1998, CANCER GENET CYTOGEN, V104, P77
   JONES AW, 2002, J ANAL TOXICOL, V26, P2
   JONES AW, 2003, FORENSIC SCI INT, V133, P1
   KORN D, 2002, ACAD MED, V77, P1089
   KURMIS AP, 2003, J BONE JOINT SURG  A, V85, P2449
   LANKHORST GJ, 2001, CLIN REHABIL, V15, P115
   LEE KP, 2002, JAMA-J AM MED ASSOC, V287, P2805
   OJASOO T, 2002, PRESSE MED, V31, P775
   OPTHOF T, 1997, CARDIOVASC RES, V33, P1
   PEARSON H, 2003, NATURE, V426, P222
   ROSTAMIHODJEGAN A, 2001, BRIT J CLIN PHARMACO, V51, P111
   SAHA S, 2003, J MED LIBR ASSOC, V91, P42
   SEGLEN PO, 1997, BRIT MED J, V314, P497
   SINDERMANN CJ, 2001, WINNING GAMES SCI PL
   SWIFT L, 2001, QUANTITATIVE METHODS
   TUGWELL P, 2004, BRIT MED J, V328, P597
   VANDIEST PJ, 2001, J CLIN PATHOL, V54, P817
   WEALE AR, 2004, BMC MED RES METHODOL, V4, P14