Peer Review Scandals

Mon Jul 21 19:41:16 EDT 2014

Davis & Stephen,

I come from a physics background, and much what you discuss bears little
resemblance to the facts on the ground. No one given a manuscript would
ever try to replicate the results. It could take months of full time work,
and is simply an unreasonable burden on the reviewer. Science would just
stop.

The purpose of peer-review (and yes, I use that term in the sense of an
expert reading and evaluating a manuscript for publication, as this is the
common vernacular in my field) is not to ensure that the results are
correct. Instead, peer review seeks to determine if the paper has obvious
mistakes or omissions, if it is original, and that if it is relevant to the
journal. It also checks that the conclusions made by the authors are
supported by the evidence presented, and that the work is presented in a
way the other experts in the field will understand. And that's about it.
And, to my mind, that's all it should be.

For a philosophical point of view, I can't imagine that journals are meant
to be repository of the one and true knowledge. Journals are discussions
amongst experts. And experts disagree. If the answer really were known,
then there'd be no reason for the journal, it'd be on Wikipedia. Journals
are a place where experts can exchange methods, results, opinions,
calculations, observations, and the like. I don't read a journal article as
gospel, but rather as a document where I cast the onus on the authors to
convince me of something. Often I'm not convinced. There are entire little
subfields that, in my opinion, are founded on flawed assumptions, and
therefore the conclusions reached are dubious. I recognise that that is
just my opinion, and I don't begrudge them (too much) for the work they do,
because I realise that I may be wrong.

As for the issue of fraud, peer review is not the place to catch it, unless
it is rather inartfully done. How can say that an observation wasn't made?
How can I say that the result of a detailed calculation is wrong? I'm not
going to do it myself. As a reviewer, I can point out the objections and
concerns of a expert, perhaps many that the authors had yet to consider,
but at the end of the day, it is their contribution to the conversation. If
they have something interesting to say, can explain it in a reasonable way,
and are sufficiently familiar with the state of field to discuss it
intelligently, who am I to say they are wrong? That is for the community to
decide. And, in my opinion, that is how science should work.

Best regards,

Dowman

On 21 July 2014 16:15, Dowman P Varn <dpv at complexmatter.org> wrote:

> David,
>
> I used the example of cold fusion because it was well-known. However, let
> me consider a much less well-known example, from my own field of research.
> It was thought for thirty years that a faulting mechanism in SiC was due to
> something called "layer-displacement faulting." Within the past few years,
> another group, with vastly improved experimental equipment and analysis
> techniques challenged this conclusion. They were able to show, both
> experimentally and by subsequent analysis, that the original conclusions
> were based on unfounded assumptions, and they could explain both the
> original data on which the 30 years old conclusions were based, as well as
> new data. The older analysis could not explain the newer results. They
> contacted the original experimental group to confirm several aspects of the
> original experiments, and subsequently were able to give a very convincing
> narrative that supported their contention that "deformation faulting" was
> the primary route to disorder in these materials under these experimental
> conditions. Based on a single paper, I, as well as the original
> researchers, are convinced that the early work had significant procedural
> flaws, and the new analysis is correct. A single paper. Well done, well
> researched, well argued, and it supplants previous work. One paper can be
> enough.
>
> It is unlikely that many experiments will get repeated hundreds of times
> by various groups before the experts reach a conclusion. Cold fusion is an
> exception. To ask so much of science is set unrealistic goals.
>
> Now, there is few philosophical point here. I am a Bayesian. I am happy to
> say that something is known not to an absolute certainty, but instead to a
> very high one. (Or even that the evidence is split amongst several
> viewpoints, and a conclusion is not justified.) And I'm always open to
> additional results, experiments, analyses and arguments that may contradict
> my world view. I happily update my prior expectations based on new
> information. And I think that this is how science should work.
>
> Best regards,
>
> Dowman
>
>
> On 18 July 2014 12:30, David Wojick <dwojick at craigellachie.us> wrote:
>
>> Adminstrative info for SIGMETRICS (for example unsubscribe):
>> http://web.utk.edu/~gwhitney/sigmetrics.html
>> Dowman, I do not consider a single failure of replication to be
>> compelling evidence against the original research, far from it. Especially
>> not if the procedure is complex, subtle or delicate. I was involved in the
>> cold fusion case and there were hundreds of attempts at replication by
>> different research groups. When they all failed the judgement became clear,
>> but only then.
>>
>> My best regards,
>>
>> David
>>
>>
>> At 03:08 PM 7/18/2014, you wrote:
>>
>> Adminstrative info for SIGMETRICS (for example unsubscribe):
>> http://web.utk.edu/~gwhitney/sigmetrics.html
>> David,
>>
>> Your first point is spot-on, but I must take issue with the second.
>> Repeatability of experimental work is a cornerstone of science. While some
>> observations are inherently unrepeatable, for example the discovery of a
>> rare fossil, others are not. For example, the cold fusion debacle of the
>> late 80's and early 90's revolved around the inability of numerous other
>> groups to repeat the experiment. While unrepeatability may be due to
>> factors other than a mistake in the original research, we'd be foolish not
>> to consider failed efforts to repeat an experiment as evidence of error in
>> the original. Perhaps not decisive evidence, but compelling evidence
>> nonetheless.
>>
>> Regards,
>>
>> Dowman
>>
>>
>> On 18 July 2014 11:55, David Wojick <dwojick at craigellachie.us > wrote:
>>  Adminstrative info for SIGMETRICS (for example unsubscribe):
>>  http://web.utk.edu/~gwhitney/sigmetrics.html
>>
>> This is a common confusion. A typical peer review takes a few hours
>> because it just involves reading the paper. The primary objective is to say
>> whether the results are important enough to publish in the reviewing
>> journal. Replication means repeating the research, which may take days,
>> weeks, months or more, depending on the project. Reading and research are
>> very different things, hence so are review and replication..
>>
>> As for your second claim, failure to replicate does not show that the
>> original research is unsound. This is another common confusion. There may
>> be a lot of procedural subtlety in the original research, which is not
>> conveyed in the journal article, which is very brief. As a result the
>> replication attempt may fail simply because something was done differently.
>> This has been discussed at length at The Scholarly Kitchen. My wife
>> recently pointed out an amusing example from baking, which is applied
>> chemistry. Forty people each made an angel food cake from the same recipe
>> and all the resulting cakes had in common was that each had a hole in the
>> middle. Journal articles seldom provide even a recipe, so failure to
>> replicate is not telling.
>>
>> David Wojick
>>  http://insidepublicaccess.com/
>>  http://scholarlykitchen.sspnet.org/author/dwojick/
>>
>>
>> At 02:31 PM 7/18/2014, you wrote:
>>  Adminstrative info for SIGMETRICS (for example unsubscribe):
>>  http://web.utk.edu/~gwhitney/sigmetrics.html
>>
>> David Wojick claimed:
>>
>> |--------------------------------------------------------------------------|
>> |"[. . .] Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â
>> Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â |
>> | Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â
>> Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â |
>> |Of course peer review has nothing to do with replication." Â  Â  Â  Â
>> Â  Â  Â  Â |
>>
>> |--------------------------------------------------------------------------|
>>
>> It is dubious to claim that being approved by reviewers should not
>> involve replication.
>>
>>
>> |---------------------------------------------------------------------------|
>> |"My guess is there are between 5 and 10 million peer reviews a year, but
>> it|
>> |only takes 4 or 5 anecdotes, some way off base, to generate broad claims
>> Â  |
>> |of wholesale corruption, that is hurting science. This is what social Â
>> Â  Â |
>> |movements feed on, and there is plenty to go around. Â  Â  Â  Â  Â  Â
>> Â  Â  Â  Â  Â  |
>> | Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â
>> Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  |
>> |[. . .]" Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â
>> Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  |
>>
>>
>> |---------------------------------------------------------------------------|
>>
>> Lack of replication harms science.
>>
>> Regards,
>> C. Gloster
>>
>>
>>
>>
>> --
>> ______________________________
>> Dowman P Varn, PhD
>> Complexity Sciences Center & Department of Physics Â  Â  Â  Â  Â  Â  Â
>> Â  Â  Â  Â  Â  Â  Â  Â
>> University of California,Â One Shields Ave
>> Davis, CA 95616
>>
>> Cell:Â 646.228.7256 Â <http://??>
>> Email: dpv at complexmatter.org
>> Web site: wissenplatz.org
>>
>
>
>
> --
> ______________________________
> Dowman P Varn, PhD
> Complexity Sciences Center & Department of Physics
>
> University of California, One Shields Ave
> Davis, CA 95616
>
> Cell: 646.228.7256
> Email: dpv at complexmatter.org
> Web site: wissenplatz.org
>

-- 
______________________________
Dowman P Varn, PhD
Complexity Sciences Center & Department of Physics

University of California, One Shields Ave
Davis, CA 95616

Cell: 646.228.7256
Email: dpv at complexmatter.org
Web site: wissenplatz.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140721/a9a03b8f/attachment.html>