Peer Review Scandals

[David Wojick]

Dear Steve,

First of all, replication also requires collecting the data, not just 
checking how it is processed. There is a great deal more to replication 
than checking someone's math. Replication means repeating the research. 
(You may be confusing replication with an audit, but even audits are very 
laborious.) It would be a colossal waste of time and a great drain on 
science if people stopped doing research in order to do replication, as you 
are calling for. Repeating every piece of research prior to publication 
basically cuts the amount of original research done in half. Is that what 
you are proposing?

I see neither a technical nor a financial necessity for open access. It 
that were true we would not need mandates, manifestos, boycotts, op-eds, 
etc. It is a classic social movement. NIH is a political organization as 
well as a research one. The Director is a political appointee. He has to do 
something in response to the political pressure of the movement.

David

At 03:54 PM 7/18/2014, you wrote:
>Adminstrative info for SIGMETRICS (for example unsubscribe):
>http://web.utk.edu/~gwhitney/sigmetrics.html
>
>As I have made abundantly clear, I do not think that open access is a 
>"movement" but a technical and financial necessity.  There is a need for 
>replication, or otherwise NIH would not make data access a 
>requirement..  What would you say to combining the two functions--an 
>article could not be published unless its research can be replicated?  In 
>other words, with the data easily accessible, editors could send the 
>article for replication before publishing.  No publication without 
>replication.  Almost sounds like a revolutionary chant you can shout in 
>the street while rioting.
>
>Steve B.
>
>-----Original Message-----
>From: ASIS&T Special Interest Group on Metrics 
>[mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of David Wojick
>Sent: Friday, July 18, 2014 2:25 PM
>To: SIGMETRICS at LISTSERV.UTK.EDU
>Subject: Re: [SIGMETRICS] Peer Review Scandals
>
>Adminstrative info for SIGMETRICS (for example unsubscribe):
>http://web.utk.edu/~gwhitney/sigmetrics.html
>
>Sorry Stephen, but I do not believe that a lot of data is being cooked.
>Some certainly, but that is to be expected when millions of people are 
>involved. I read this as a false scare being promulgated by people with an 
>agenda, namely the "open" movement.
>
>David
>
>At 03:07 PM 7/18/2014, you wrote:
> >Adminstrative info for SIGMETRICS (for example unsubscribe):
> >http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> >Needless to say, lack of ability to replicate is one of the main
> >reasons for financial losses in the pharmaceutical industry, sending
> >many companies down wrong paths.  The problem is so bad that it is my
> >understanding that the NIH is requiring that data be made openly
> >accessible if the research is the result of an NIH grant so that it can
> >be replicated.  This is one of the main reasons why universities are
> >establishing open access institutional repositories.  LSU is
> >considering the Dataverse Network, on which ASIST just presented a
> >Webinar.  If interested, see the URL below:
> >
> >   http://en.wikipedia.org/wiki/Dataverse
> >
> >There is more than cakes being cooked in various kitchens.  There is a
> >lot of data also being cooked.
> >
> >Stephen J. Bensman
> >LSU Libraries
> >USA
> >
> >-----Original Message-----
> >From: ASIS&T Special Interest Group on Metrics
> >[mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of David Wojick
> >Sent: Friday, July 18, 2014 1:56 PM
> >To: SIGMETRICS at LISTSERV.UTK.EDU
> >Subject: Re: [SIGMETRICS] Peer Review Scandals
> >
> >Adminstrative info for SIGMETRICS (for example unsubscribe):
> >http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> >This is a common confusion. A typical peer review takes a few hours
> >because it just involves reading the paper. The primary objective is to
> >say whether the results are important enough to publish in the
> >reviewing journal.
> >Replication means repeating the research, which may take days, weeks,
> >months or more, depending on the project. Reading and research are very
> >different things, hence so are review and replication..
> >
> >As for your second claim, failure to replicate does not show that the
> >original research is unsound. This is another common confusion. There
> >may be a lot of procedural subtlety in the original research, which is
> >not conveyed in the journal article, which is very brief. As a result
> >the replication attempt may fail simply because something was done 
> differently.
> >This has been discussed at length at The Scholarly Kitchen. My wife
> >recently pointed out an amusing example from baking, which is applied
> >chemistry. Forty people each made an angel food cake from the same
> >recipe and all the resulting cakes had in common was that each had a
> >hole in the middle. Journal articles seldom provide even a recipe, so
> >failure to replicate is not telling.
> >
> >David Wojick
> >http://insidepublicaccess.com/
> >http://scholarlykitchen.sspnet.org/author/dwojick/
> >
> >At 02:31 PM 7/18/2014, you wrote:
> > >Adminstrative info for SIGMETRICS (for example unsubscribe):
> > >http://web.utk.edu/~gwhitney/sigmetrics.html
> > >
> > >David Wojick claimed:
> > >|---------------------------------------------------------------------- 
> ----|
> > >|"[. . 
> .]                                                                  |
> > >| 
>      |
> > >|Of course peer review has nothing to do with 
> replication."                |
> > >|---------------------------------------------------------------------- 
> ----|
> > >
> > >It is dubious to claim that being approved by reviewers should not
> > >involve replication.
> > >
> > >|--------------------------------------------------------------------
> > >|----
> > ---|
> > >|"My guess is there are between 5 and 10 million peer reviews a year,
> > but it|
> > >|only takes 4 or 5 anecdotes, some way off base, to generate broad
> > claims   |
> > >|of wholesale corruption, that is hurting science. This is what
> > social      |
> > >|movements feed on, and there is plenty to go
> > around.                       |
> > >|
> >     |
> > >|[. .
> > .]"                                                                   |
> > >|--------------------------------------------------------------------
> > >|----
> > ---|
> > >
> > >Lack of replication harms science.
> > >
> > >Regards,
> > >C. Gloster