Peer Review Scandals

[Dowman P Varn]

David,

I used the example of cold fusion because it was well-known. However, let
me consider a much less well-known example, from my own field of research.
It was thought for thirty years that a faulting mechanism in SiC was due to
something called "layer-displacement faulting." Within the past few years,
another group, with vastly improved experimental equipment and analysis
techniques challenged this conclusion. They were able to show, both
experimentally and by subsequent analysis, that the original conclusions
were based on unfounded assumptions, and they could explain both the
original data on which the 30 years old conclusions were based, as well as
new data. The older analysis could not explain the newer results. They
contacted the original experimental group to confirm several aspects of the
original experiments, and subsequently were able to give a very convincing
narrative that supported their contention that "deformation faulting" was
the primary route to disorder in these materials under these experimental
conditions. Based on a single paper, I, as well as the original
researchers, are convinced that the early work had significant procedural
flaws, and the new analysis is correct. A single paper. Well done, well
researched, well argued, and it supplants previous work. One paper can be
enough.

It is unlikely that many experiments will get repeated hundreds of times by
various groups before the experts reach a conclusion. Cold fusion is an
exception. To ask so much of science is set unrealistic goals.

Now, there is few philosophical point here. I am a Bayesian. I am happy to
say that something is known not to an absolute certainty, but instead to a
very high one. (Or even that the evidence is split amongst several
viewpoints, and a conclusion is not justified.) And I'm always open to
additional results, experiments, analyses and arguments that may contradict
my world view. I happily update my prior expectations based on new
information. And I think that this is how science should work.

Best regards,

Dowman


On 18 July 2014 12:30, David Wojick <dwojick at craigellachie.us> wrote:

> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
> Dowman, I do not consider a single failure of replication to be compelling
> evidence against the original research, far from it. Especially not if the
> procedure is complex, subtle or delicate. I was involved in the cold fusion
> case and there were hundreds of attempts at replication by different
> research groups. When they all failed the judgement became clear, but only
> then.
>
> My best regards,
>
> David
>
>
> At 03:08 PM 7/18/2014, you wrote:
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
> David,
>
> Your first point is spot-on, but I must take issue with the second.
> Repeatability of experimental work is a cornerstone of science. While some
> observations are inherently unrepeatable, for example the discovery of a
> rare fossil, others are not. For example, the cold fusion debacle of the
> late 80's and early 90's revolved around the inability of numerous other
> groups to repeat the experiment. While unrepeatability may be due to
> factors other than a mistake in the original research, we'd be foolish not
> to consider failed efforts to repeat an experiment as evidence of error in
> the original. Perhaps not decisive evidence, but compelling evidence
> nonetheless.
>
> Regards,
>
> Dowman
>
>
> On 18 July 2014 11:55, David Wojick <dwojick at craigellachie.us > wrote:
>  Adminstrative info for SIGMETRICS (for example unsubscribe):
>  http://web.utk.edu/~gwhitney/sigmetrics.html
>
> This is a common confusion. A typical peer review takes a few hours
> because it just involves reading the paper. The primary objective is to say
> whether the results are important enough to publish in the reviewing
> journal. Replication means repeating the research, which may take days,
> weeks, months or more, depending on the project. Reading and research are
> very different things, hence so are review and replication..
>
> As for your second claim, failure to replicate does not show that the
> original research is unsound. This is another common confusion. There may
> be a lot of procedural subtlety in the original research, which is not
> conveyed in the journal article, which is very brief. As a result the
> replication attempt may fail simply because something was done differently.
> This has been discussed at length at The Scholarly Kitchen. My wife
> recently pointed out an amusing example from baking, which is applied
> chemistry. Forty people each made an angel food cake from the same recipe
> and all the resulting cakes had in common was that each had a hole in the
> middle. Journal articles seldom provide even a recipe, so failure to
> replicate is not telling.
>
> David Wojick
>  http://insidepublicaccess.com/
>  http://scholarlykitchen.sspnet.org/author/dwojick/
>
>
> At 02:31 PM 7/18/2014, you wrote:
>  Adminstrative info for SIGMETRICS (for example unsubscribe):
>  http://web.utk.edu/~gwhitney/sigmetrics.html
>
> David Wojick claimed:
>
> |--------------------------------------------------------------------------|
> |"[. . .] Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â
> Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â |
> | Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â
> Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â |
> |Of course peer review has nothing to do with replication." Â  Â  Â  Â  Â
> Â  Â  Â |
>
> |--------------------------------------------------------------------------|
>
> It is dubious to claim that being approved by reviewers should not
> involve replication.
>
>
> |---------------------------------------------------------------------------|
> |"My guess is there are between 5 and 10 million peer reviews a year, but
> it|
> |only takes 4 or 5 anecdotes, some way off base, to generate broad claims
> Â  |
> |of wholesale corruption, that is hurting science. This is what social Â
> Â  Â |
> |movements feed on, and there is plenty to go around. Â  Â  Â  Â  Â  Â  Â
> Â  Â  Â  Â  |
> | Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â
> Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  |
> |[. . .]" Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â
> Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  Â  |
>
>
> |---------------------------------------------------------------------------|
>
> Lack of replication harms science.
>
> Regards,
> C. Gloster
>
>
>
>
> --
> ______________________________
> Dowman P Varn, PhD
> Complexity Sciences Center & Department of Physics               Â
> Â  Â  Â  Â  Â  Â  Â
> University of California, One Shields Ave
> Davis, CA 95616
>
> Cell:Â 646.228.7256 Â <http://??>
> Email: dpv at complexmatter.org
> Web site: wissenplatz.org
>



-- 
______________________________
Dowman P Varn, PhD
Complexity Sciences Center & Department of Physics

University of California, One Shields Ave
Davis, CA 95616

Cell: 646.228.7256
Email: dpv at complexmatter.org
Web site: wissenplatz.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140721/1191bee3/attachment.html>