Takeo Nakayama, Tsuguya Fukui, Shunichi Fukuhara,Kiichiro Tsutani, Shigeaki Yamazaki,"Comparison Between Impact Factors and Citations in Evidence-Based Practice Guidelines" JAMA 290(6):755-756, August 13 2003

[Eugene Garfield]

Takeo Nakayama : nakayama at pbh.med.kyoto-u.ac.jp

REPRINTED WITH PERMISSION FROM THE AUTHOR

Title    Comparison Between Impact Factors and Citations in Evidence-Based
         Practice Guidelines

Journal  JAMA. 290(6):755-756. August 13, 2003

Authors  Takeo Nakayama, Tsuguya Fukui, Shunichi Fukuhara,
         Kiichiro Tsutani, Shigeaki Yamazaki,

To the Editor: Impact factors of medical journals are calculated as the
total number of current citations of articles published in a journal during
the previous 2 calendar years divided by the total number of designated
articles published in that journal during the same period.1 Thus, impact
factors indicate the annual average number of citations of articles that
have appeared in a given journal.

Impact factors are widely regarded as a quality ranking for scientific
journals. Concerns have arisen, however, that scientific communities might
be overly reliant on impact factors to assess the worth of scientific
publications, as these numbers may be artifically inflated in a number of
ways.2-4 A related problem is that entire scientific disciplines tend to be
evaluated based on average impact factors of their collective journals.5
Despite these concerns, researchers who have published in journals
with high impact factors may be more likely to be rewarded by their
institutions.

We attempted to validate this interpretation of impact factors by analyzing
the bibliographic citations of articles used to support the Guide to
Clinical Preventive Services6 by the US Preventive Services Task Force
(USPSTF). The USPSTF guidelines are generally thought to reflect the highest
level of scientific evidence. Thus, we hypothesized that the guidelines
should be supported by a larger number of citations from journals with high
impact factors.

Methods
We assessed all references in all 25 chapters of the current version of the
USPSTF guidelines and counted the number of references from each journal
listed. The journals' impact factors for 2001 were obtained from the
Institute for Scientific Information and the Journal of Citation Reports
(Science/Social Science; version for 2001).7 Counts were recombined for
journals that were renamed.

Results
Among the total 1740 citations in the reference sections of the 25 chapters,
1531 were from scientific journals and 209 were from academic books and
official reports. The most cited journal was JAMA with 135 citations,
followed by the American Journal of Preventive Medicine (102), BMJ (77), and
The Lancet (70). Fifty-six journals had articles cited more than 5 times,
comprising a total of 1185 citations. Of these 56 journals, 6 (11%) had an
impact factor of more than 10.0; 10 (18%) had an impact factor of 5.0 to
10.0; 11 (20%) had an impact factor of 3.0 to 5.0; and 28 (51%) had impact
factors of less than 3.0. Of this latter group, 11 20%) had impact factors
of less than 2.0. Only 7 journals (13%) appeared in the top 100 journals
ranked by impact factors (2001). The median impact factor of these 56
journals was 2.76. There was a significant correlation between impact
factors and times cited in the USPSTF guidelines (Kendall r = 0.26, P =
.005).

Comment
We found that the number of citations by the USPSTF guidelines roughly
parallels the impact factors for the respective journals. Journals with low
impact factors, however, were also cited frequently as providing important
evidence. This finding may reflect the fact that journals that focus on
preventive services tend to have lower impact factors than do journals in
other scientific disciplines.

Some of the possible domains of impact of journal articles that cannot be
measured by impact factors are changes in readers' knowledge,
practice, clinical outcomes, funding priorities for research, and prompting
of further learning. Overreliance on impact factors may undervalue the
unique contributions of individual areas of research. In the field of
clinical or preventive medicine, in particular, citation analyses on
evidence-based practice guidelines may be a more accurate assessment of the
contributions of individual journals and researchers. Although we
only assessed the area of preventive health services, we suspect that this
general conclusion may extend to other areas of scientific inquiry.

Acknowledgment: We thank Ms Akiko Yoshida for assistance in editing the
manuscript.

Takeo Nakayama, MD, PhD
Department of Medical System Informatics

Tsuguya Fukui, MD, PhD
Department of General Medicine and Clinical Epidemiology

Shunichi Fukuhara, MD, DMsc
Department of Healthcare Research
Kyoto University Graduate School of Medicine
Kyoto, Japan

Kiichiro Tsutani, MD, PhD
Department of Pharmacoeconomics
Graduate School of Pharmaceutical Sciences
The University of Tokyo
Tokyo, Japan

Shigeaki Yamazaki, PhD
Department of Library and Information Science
Aichi Shukutoku University
Aichi, Japan


1. Garfield E. Which medical journals have the greatest impact? Ann Intern
Med. 1986;105:313-320. ISI | MEDLINE

2. Seglen PO. Why the impact factor of journals should not be used for
evaluating research? BMJ. 1997;314:498-502. FULL TEXT

3. Smith R. Journal accused of manipulating impact factor. BMJ.
1997;314:463.

4. Hemmingsson A, Mygind T, Skjennald A, Edgren J. Manipulation of impact
factors by editors of scientific journals. AJR Am J Roentgenol.
2002;178:767. FULL TEXT

5. Valdecasas AG, Castroviejo S, Marcus LF. Reliance on the citation index
undermines the study of biodiversity. Nature. 2000;403:698. MEDLINE

6. US Preventive Services Task Force. Guide to clinical preventive services,
second edition. Available at:
http://www.odphp.osophs.dhhs.gov/pubs/guidecps/. Accessed June 25, 2003.

7. ISI Web of Knowledge. Available at:
http://www.isinet.com/isi/products/citation/jcr/jcrweb. Accessibility
verified (paid subscription) July 24,
2003.

When responding, please attach my original message
_______________________________________________________________________
Eugene Garfield, PhD.  email: garfield at codex.cis.upenn.edu
home page: www.eugenegarfield.org
Tel: 215-243-2205 Fax 215-387-1266
President, The Scientist LLC. www.the-scientist.com
Chairman Emeritus, ISI www.isinet.com
Past President, American Society for Information Science and Technology
(ASIS&T)  www.asis.org
_______________________________________________________________________