Fu LD, Aphinyanaphongs Y, Wang LL, Aliferis CF "A comparison of evaluation metrics for biomedical journals, articles, and websites in terms of sensitivity to topic " JASIST 44(4):587-594, August 2011

Thu Aug 25 16:43:33 EDT 2011

E-mail: lawrence.fu at nyumc.org

TITLE  :A comparison of evaluation metrics for biomedical journals, articles, 
and websites in terms of sensitivity to topic  

Author(s): Fu, LD (Fu, Lawrence D.)1; Aphinyanaphongs, Y 
(Aphinyanaphongs, Yindalon)1; Wang, LL (Wang, Lily)2; Aliferis, CF 
(Aliferis, Constantin F.)1  

Source: JOURNAL OF BIOMEDICAL INFORMATICS  Volume: 44   Issue: 4   
Pages: 587-594   DOI: 10.1016/j.jbi.2011.03.006   Published: AUG 2011  

Times Cited: 0 (from Web of Science)  

Cited References: 22 [ view related records ]     Citation Map      

Abstract: 
Evaluating the biomedical literature and health-related websites for quality 
are challenging information retrieval tasks. Current commonly used methods 
include impact factor for journals, PubMed's clinical query filters and machine 
learning-based filter models for articles, and PageRank for websites. Previous 
work has focused on the average performance of these methods without 
considering the topic, and it is unknown how performance varies for specific 
topics or focused searches. Clinicians, researchers, and users should be 
aware when expected performance is not achieved for specific topics. The 
present work analyzes the behavior of these methods for a variety of topics. 
Impact factor, clinical query filters, and PageRank vary widely across 
different topics while a topic-specific impact factor and machine learning-
based filter models are more stable. The results demonstrate that a method 
may perform excellently on average but struggle when used on a number of 
narrower topics. Topic-adjusted metrics and other topic robust methods have 
an advantage in such situations. Users of traditional topic-sensitive metrics 
should be aware of their limitations. (C) 2011 Elsevier Inc. All rights 
reserved.  

Document Type: Article  
Language: English  

Author Keywords: Information retrieval; Machine learning; PageRank; 
Journal impact factor; Topic-sensitivity; Bibliometrics  
KeyWords Plus: IMPACT FACTORS; PAGERANK; SEARCH; LINK  

Reprint Address: Fu, LD (reprint author), NYU, Med Ctr, Ctr Hlth Informat & 
Bioinformat, 227 E 30th St,7th Floor, New York, NY 10016 USA  
Addresses: 
1. NYU, Med Ctr, Ctr Hlth Informat & Bioinformat, New York, NY 10016 USA 
2. Vanderbilt Univ, Sch Med, Dept Biostat, Nashville, TN 37232 USA  

E-mail Address: 
lawrence.fu at nyumc.org 
yin.a at nyumc.org
lily.wang at vanderbilt.edu
constantin.aliferis at nyumc.org 

Funding: 
Funding Agency Grant Number 
  R56 LM007948-04A1  
  1UL1RR029893  

[Show funding text][Hide funding text]   

The authors gratefully acknowledge support from Grants R56 LM007948-
04A1 and 1UL1RR029893.

Publisher: ACADEMIC PRESS INC ELSEVIER SCIENCE, 525 B ST, STE 1900, 
SAN DIEGO, CA 92101-4495 USA  
Web of Science Category: Computer Science, Interdisciplinary Applications; 
Medical Informatics  
Subject Category: Computer Science; Medical Informatics  
IDS Number: 800OO  
ISSN: 1532-0464  

CITED REFERENCES:
1. 

 Title: [not available]
Author(s): APHINYANAPHONGS Y
Source: AMIA ANN S  Published: 2003 
Times Cited: 1 (from All Databases) 

  2. 

 Title: Text categorization models for high-quality article retrieval in internal 
medicine 

Author(s): Aphinyanaphongs Y; Tsamardinos I; Statnikov A; et al.
Source: JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION  
Volume: 12   Issue: 2   Pages: 207-216   DOI: 10.1197/jamia.M1641   
Published: MAR-APR 2005 
Times Cited: 30 (from All Databases) 

  3. 

 Title: STATISTICAL-METHODS FOR ASSESSING AGREEMENT BETWEEN 2 
METHODS OF CLINICAL MEASUREMENT 

Author(s): BLAND JM; ALTMAN DG
Source: LANCET  Volume: 1   Issue: 8476   Pages: 307-310   Published: FEB 
8 1986 
Times Cited: 18,893 (from All Databases) 

  4. 

 Title: [not available]
Author(s): BORODIN A
Source: 10 INT WORLD WID WEB  Published: 2001 
Times Cited: 1 (from All Databases) 

  5. 

 Title: [not available]
Author(s): BRIN S
Source: P 7 INT WORLD WID WE  Published: 1998 
Times Cited: 188 (from All Databases) 

  6. 

 Title: A tutorial on Support Vector Machines for pattern recognition 

Author(s): Burges CJC
Source: DATA MINING AND KNOWLEDGE DISCOVERY  Volume: 2   Issue: 2   
Pages: 121-167   DOI: 10.1023/A:1009715923555   Published: JUN 1998 
Times Cited: 2,863 (from All Databases) 

  7. 

 Title: [not available]
Author(s): GARFIELD E
Source: CAN CITATION INDEXIN  Published: 1965 
Times Cited: 4 (from All Databases) 

  8. 

 Title: CITATION ANALYSIS AS A TOOL IN JOURNAL EVALUATION - 
JOURNALS CAN BE RANKED BY FREQUENCY AND IMPACT OF CITATIONS FOR 
SCIENCE POLICY STUDIES 

Author(s): GARFIELD E
Source: SCIENCE  Volume: 178   Issue: 4060   Pages: 471-+   DOI: 
10.1126/science.178.4060.471   Published: 1972 
Times Cited: 851 (from All Databases) 

  9. 

 Title: The history and meaning of the journal impact factor 

Author(s): Garfield E
Source: JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION  
Volume: 295   Issue: 1   Pages: 90-93   DOI: 10.1001/jama.295.1.90   
Published: JAN 4 2006 
Times Cited: 308 (from All Databases) 

  10. 

 Title: Journal impact measures in bibliometric research 

Author(s): Glanzel W; Moed HF
Conference: 8th International Conference on Scientometrics and Informetrics 
Location: SYDNEY, AUSTRALIA Date: JUL 17, 2001 
Source: SCIENTOMETRICS  Volume: 53   Issue: 2   Pages: 171-193   DOI: 
10.1023/A:1014848323806   Published: FEB 2002 
Times Cited: 119 (from All Databases) 

  11. 

 Title: Topic-sensitive PageRank: A context-sensitive ranking algorithm for 
Web search 

Author(s): Haveliwala TH
Conference: 11th International World Wide Web Conference Location: 
HONOLULU, HAWAII Date: MAY 07-11, 2002 
Source: IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING  
Volume: 15   Issue: 4   Pages: 784-796   DOI: 
10.1109/TKDE.2003.1208999   Published: JUL-AUG 2003 
Times Cited: 91 (from All Databases) 

  12. 

 Title: DEVELOPING OPTIMAL SEARCH STRATEGIES FOR DETECTING 
CLINICALLY SOUND STUDIES IN MEDLINE 

Author(s): HAYNES RB; WILCZYNSKI N; MCKIBBON KA; et al.
Source: JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION  
Volume: 1   Issue: 6   Pages: 447-458   Published: NOV-DEC 1994 
Times Cited: 322 (from All Databases) 

  13. 

 Title: [not available]
Author(s): KAMVAR S
Source: EXPLOITING BLOCK STR  Published: 2003 
Times Cited: 51 (from All Databases) 

  14. 

 Title: Rank-stability and rank-similarity of link-based Web ranking 
algorithms in authority-connected graphs 

Author(s): Lempel R; Moran S
Conference: Workshop on Mathematical/Formal Methods in Information 
Retrieval Location: Tampere, FINLAND Date: 2002 
Sponsor(s): ACM SIGIR 
Source: INFORMATION RETRIEVAL  Volume: 8   Issue: 2   Pages: 245-264   
DOI: 10.1007/s10791-005-5661-0   Published: APR 2005 
Times Cited: 5 (from All Databases) 

  15. 

 Title: [not available]
Author(s): *NAT LIB MED
Source: MESH BROWS   
Times Cited: 4 (from All Databases) 

  16. 

 Title: [not available]
Author(s): NG A
Source: 24 ACM INT C RES DEV  Published: 2001 
Times Cited: 1 (from All Databases) 

  17. 

 Title: [not available]
Author(s): NIE L
Source: 29 ACM INT C RES DEV  Published: 2006 
Times Cited: 1 (from All Databases) 

  18. 

 Title: The intelligent surfer: Probabilistic combination of link and content 
information in PageRank 

Author(s): Richardson M; Domingos P
Editor(s): Dietterich TG; Becker S; Ghahramani Z
Conference: 15th Annual Conference on Neural Information Processing 
Systems (NIPS) Location: VANCOUVER, CANADA Date: DEC 03-08, 2001 
Source: ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 14, 
VOLS 1 AND 2  Book Series: ADVANCES IN NEURAL INFORMATION 
PROCESSING SYSTEMS   Volume: 14   Pages: 1441-1448   Published: 2002 
Times Cited: 31 (from All Databases) 

  19. 

 Title: [not available]
Author(s): TABACHNICK BG
Source: USING MULTIVARIATE S  Published: 2006 
Times Cited: 197 (from All Databases) 

  20. 

 Title: An alternative to journal-based impact factors 

Author(s): Takahashi K; Aw TC; Koh D
Source: OCCUPATIONAL MEDICINE-OXFORD  Volume: 49   Issue: 1   Pages: 
57-58   DOI: 10.1093/occmed/49.1.57-a   Published: JAN 1999 
Times Cited: 16 (from All Databases) 

  21. 

 Title: [not available]
Author(s): *THOMS SCI
Source: ISI WEB KNOWL   
Times Cited: 6 (from All Databases) 

  22. 

 Title: A proposal for topic-based impact factors and their application to 
occupational health literature 

Author(s): Uehara M; Takahashi K; Hoshuyama T; et al.
Source: JOURNAL OF OCCUPATIONAL HEALTH  Volume: 45   Issue: 4   
Pages: 248-253   DOI: 10.1539/joh.45.248   Published: JUL 2003 
Times Cited: 9 (from All Databases) 