Jackson JL, Srinivasan M, Rea J, Fletcher KE Kravitz RL "The validity of Peer Review in a General Medicine Journal" PLOS One 6(7) Article #e22475, July 25, 2011

[Eugene Garfield]

E-mail Address: jjackson at mcw.edu  

 
TITLE : The Validity of Peer Review in a General Medicine Journal  

Author(s): Jackson, JL (Jackson, Jeffrey L.)1; Srinivasan, M (Srinivasan, 
Malathi)2; Rea, J (Rea, Joanna)1; Fletcher, KE (Fletcher, Kathlyn E.)1; 
Kravitz, RL (Kravitz, Richard L.)2  

Source: PLOS ONE  Volume: 6   Issue: 7     Article Number: e22475   DOI: 
10.1371/journal.pone.0022475   Published: JUL 25 2011  

Times Cited: 0 (from Web of Science)  

Cited References: 16 [ view related records ]     Citation Map      

Abstract: 
All the opinions in this article are those of the authors and should not be 
construed to reflect, in any way, those of the Department of Veterans Affairs. 

Background: Our study purpose was to assess the predictive validity of 
reviewer quality ratings and editorial decisions in a general medicine journal. 

Methods: Submissions to the Journal of General Internal Medicine (JGIM) 
between July 2004 and June 2005 were included. We abstracted JGIM peer 
review quality ratings, verified the publication status of all articles and 
calculated an impact factor for published articles (Rw) by dividing the 3-year 
citation rate by the average for this group of papers; an Rw > 1 indicates a 
greater than average impact. 

Results: Of 507 submissions, 128 (25%) were published in JGIM, 331 
rejected (128 with review) and 48 were either not resubmitted after revision 
was requested or were withdrawn by the author. Of 331 rejections, 243 were 
published elsewhere. Articles published in JGIM had a higher citation rate 
than those published elsewhere (Rw: 1.6 vs. 1.1, p = 0.002). Reviewer 
quality ratings of article quality had good internal consistency and reviewer 
recommendations markedly influenced publication decisions. There was no 
quality rating cutpoint that accurately distinguished high from low impact 
articles. There was a stepwise increase in Rw for articles rejected without 
review, rejected after review or accepted by JGIM (Rw 0.60 vs. 0.87 vs. 1.56, 
p < 0.0005). However, there was low agreement between reviewers for 
quality ratings and publication recommendations. The editorial publication 
decision accurately discriminated high and low impact articles in 68% of 
submissions. We found evidence of better accuracy with a greater number of 
reviewers. 

Conclusions: The peer review process largely succeeds in selecting high 
impact articles and dispatching lower impact ones, but the process is far 
from perfect. While the inter-rater reliability between individual reviewers is 
low, the accuracy of sorting is improved with a greater number of reviewers.  

Document Type: Review  
Language: English  
KeyWords Plus: MANUSCRIPT QUALITY; AUTHORS  

Reprint Address: Jackson, JL (reprint author), Zablocki VA Med Ctr, Div Gen 
Med, Milwaukee, WI USA  

Addresses: 
1. Zablocki VA Med Ctr, Div Gen Med, Milwaukee, WI USA 
2. Univ Calif Davis, Div Gen Med, Sacramento, CA 95817 USA  

E-mail Address: jjackson at mcw.edu  

Publisher: PUBLIC LIBRARY SCIENCE, 185 BERRY ST, STE 1300, SAN 
FRANCISCO, CA 94107 USA  
Web of Science Category: Biology; Multidisciplinary Sciences  
Subject Category: Life Sciences & Biomedicine - Other Topics; Science & 
Technology - Other Topics  
IDS Number: 798AP  
ISSN: 1932-6203  
 
 CITED REFERENCES  

1. 

 Title: Effect of statistical review on manuscript quality in Medicina Clinica 
(Barcelona): a randomized study 

Author(s): Arnau C; Cobo E; Ribera JM; et al.
Source: MEDICINA CLINICA  Volume: 121   Issue: 18   Pages: 690-694   
DOI: 10.1157/13054597   Published: NOV 22 2003 
Times Cited: 8 (from All Databases) 
   
  2. 

 Title: [not available]
Author(s): BORMANN L
Source: PLOS ONE  Volume: 28   Published: 2010 
Times Cited: 1 (from All Databases) 
  
  3. 

 Title: Extent of type I and type II errors in editorial decisions: A case study 
on Angewandte Chemie International Edition 

Author(s): Bornmann Lutz; Daniel Hans-Dieter
Source: JOURNAL OF INFORMETRICS  Volume: 3   Issue: 4   Pages: 348-
352   DOI: 10.1016/j.joi.2009.05.002   Published: OCT 2009 
Times Cited: 8 (from All Databases) 
   
  4. 

 Title: Reliability of editors' subjective quality ratings of peer reviews of 
manuscripts 

Author(s): Callaham ML; Baxt WG; Waeckerie JF; et al.
Conference: 3rd International Congress on Peer Review in Biomedical 
Publication Location: PRAGUE, CZECH REPUBLIC Date: SEP, 1997 
Sponsor(s): Journal of the Amer Med Assoc; British Med J; Project HOPE 
Source: JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION  
Volume: 280   Issue: 3   Pages: 229-231   DOI: 10.1001/jama.280.3.229   
Published: JUL 15 1998 
Times Cited: 40 (from All Databases) 
   
  5. 

 Title: The use of dedicated methodology and statistical reviewers for peer 
review: A content analysis of comments to authors made by methodology 
and regular reviewers 

Author(s): Day FC; Schriger DL; Todd C; et al.
Conference: 4th International Congress on Peer Review in Biomedical 
Publication Location: BARCELONA, SPAIN Date: SEP 14-16, 2001 
Source: ANNALS OF EMERGENCY MEDICINE  Volume: 40   Issue: 3   Pages: 
329-333   DOI: 10.1067/mem.2002.127326   Published: SEP 2002 
Times Cited: 13 (from All Databases) 
   
  6. 

 Title: MANUSCRIPT QUALITY BEFORE AND AFTER PEER-REVIEW AND 
EDITING AT ANNALS OF INTERNAL-MEDICINE 

Author(s): GOODMAN SN; BERLIN J; FLETCHER SW; et al.
Source: ANNALS OF INTERNAL MEDICINE  Volume: 121   Issue: 1   Pages: 
11-21   Published: JUL 1 1994 
Times Cited: 120 (from All Databases) 
   
  7. 

 Title: What is submitted and what gets accepted in Indian Pediatrics: 
Analysis of submissions, review process, decision making, and criteria for 
rejection 

Author(s): Gupta P; Kaur G; Sharma B; et al.
Source: INDIAN PEDIATRICS  Volume: 43   Issue: 6   Pages: 479-489   
Published: JUN 2006 
Times Cited: 8 (from All Databases) 
   
  8. 

 Title: [not available]
Author(s): JEFFERSON T
Source: COCHRANE DB SYST REV  Published: 2008 
Times Cited: 1 (from All Databases) 
  
  9. 

 Title: EFFECT SIZES FOR INTERPRETING CHANGES IN HEALTH-STATUS 

Author(s): KAZIS LE; ANDERSON JJ; MEENAN RF
Source: MEDICAL CARE  Volume: 27   Issue: 3   Supplement: S   Pages: 
S178-S189   DOI: 10.1097/00005650-198903001-00015   Published: MAR 
1989 
Times Cited: 579 (from All Databases) 
   
  10. 

 Title: [not available]
Author(s): KRAVITZ RL
Source: PLOS ONE  Volume: 8   Published: 2010 
Times Cited: 1 (from All Databases) 
  
  11. 

 Title: [not available]
Author(s): LOCK S
Source: DIFFICULT BALANCE ED  Published: 1985 
Times Cited: 130 (from All Databases) 
  
  12. 

 Title: Peer review in the Croatian Medical Journal from 1992 to 1996. 

Author(s): Marusic A; Mestrovic T; Petrovecki M; et al.
Source: Croatian medical journal  Volume: 39   Issue: 1   Pages: 3-9   
Published: 1998-Mar 
Times Cited: 11 (from All Databases) 
  
  13. 

 Title: Readers' evaluation of effect of peer review and editing on quality of 
articles in the Nederlands Tijdschrift voor Geneeskunde 

Author(s): Pierie JPEN; Walvoort HC; Overbeke AJPM
Source: LANCET  Volume: 348   Issue: 9040   Pages: 1480-1483   DOI: 
10.1016/S0140-6736(96)05016-7   Published: NOV 30 1996 
Times Cited: 41 (from All Databases) 
   
  14. 

 Title: Authors' and Editors' Perspectives on Peer Review Quality in Three 
Scholarly Nursing Journals 

Author(s): Shattell Mona M.; Chinn Peggy; Thomas Sandra P.; et al.
Source: JOURNAL OF NURSING SCHOLARSHIP  Volume: 42   Issue: 1   
Pages: 58-65   DOI: 10.1111/j.1547-5069.2009.01331.x   Published: 2010 
Times Cited: 3 (from All Databases) 
   
  15. 

 Title: Relations of relative scientometric impact indicators. The relative 
publication strategy index 

Author(s): Vinkler P

Source: SCIENTOMETRICS  Volume: 40   Issue: 1   Pages: 163-169   DOI: 
10.1007/BF02459266   Published: SEP 1997 
Times Cited: 37 (from All Databases) 
   
  16. 

 Title: Editorial peer review: A comparison of authors publishing in two 
groups of US medical journals 

Author(s): Weller AC
Source: BULLETIN OF THE MEDICAL LIBRARY ASSOCIATION  Volume: 84   
Issue: 3   Pages: 359-366   Published: JUL 1996 

Times Cited: 10 (from All Databases)