Peer Review Scandals

[Stephen J Bensman]

Needless to say, lack of ability to replicate is one of the main reasons for financial losses in the pharmaceutical industry, sending many companies down wrong paths.  The problem is so bad that it is my understanding that the NIH is requiring that data be made openly accessible if the research is the result of an NIH grant so that it can be replicated.  This is one of the main reasons why universities are establishing open access institutional repositories.  LSU is considering the Dataverse Network, on which ASIST just presented a Webinar.  If interested, see the URL below:

  http://en.wikipedia.org/wiki/Dataverse

There is more than cakes being cooked in various kitchens.  There is a lot of data also being cooked.

Stephen J. Bensman
LSU Libraries
USA

-----Original Message-----
From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of David Wojick
Sent: Friday, July 18, 2014 1:56 PM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] Peer Review Scandals


This is a common confusion. A typical peer review takes a few hours because it just involves reading the paper. The primary objective is to say whether the results are important enough to publish in the reviewing journal. 
Replication means repeating the research, which may take days, weeks, months or more, depending on the project. Reading and research are very different things, hence so are review and replication..

As for your second claim, failure to replicate does not show that the original research is unsound. This is another common confusion. There may be a lot of procedural subtlety in the original research, which is not conveyed in the journal article, which is very brief. As a result the replication attempt may fail simply because something was done differently. 
This has been discussed at length at The Scholarly Kitchen. My wife recently pointed out an amusing example from baking, which is applied chemistry. Forty people each made an angel food cake from the same recipe and all the resulting cakes had in common was that each had a hole in the middle. Journal articles seldom provide even a recipe, so failure to replicate is not telling.

David Wojick
http://insidepublicaccess.com/
http://scholarlykitchen.sspnet.org/author/dwojick/

At 02:31 PM 7/18/2014, you wrote:
>Adminstrative info for SIGMETRICS (for example unsubscribe):
>http://web.utk.edu/~gwhitney/sigmetrics.html
>
>David Wojick claimed:
>|--------------------------------------------------------------------------|
>|"[. . .]                                                                  |
>|                                                                          |
>|Of course peer review has nothing to do with replication."                |
>|--------------------------------------------------------------------------|
>
>It is dubious to claim that being approved by reviewers should not 
>involve replication.
>
>|---------------------------------------------------------------------------|
>|"My guess is there are between 5 and 10 million peer reviews a year, but it|
>|only takes 4 or 5 anecdotes, some way off base, to generate broad claims   |
>|of wholesale corruption, that is hurting science. This is what social      |
>|movements feed on, and there is plenty to go around.                       |
>|                                                                           |
>|[. . .]"                                                                   |
>|---------------------------------------------------------------------------|
>
>Lack of replication harms science.
>
>Regards,
>C. Gloster