Peer Review Scandals

Tue Jul 15 14:30:35 EDT 2014

Needless to say, it has provoked a lot of commentary-353 comments in one day.  Three of them are below.  It is an op-ed piece in a newspaper with a certain agenda, so do not expect too much.

Stephen J Bensman, Ph.D.
LSU Libraries
Lousiana State University
Baton Rouge, LA 70803

1.       This entire discussion, including the articles in the NY Times, Physics Today, and the Economist are based on a faulty, one might even say fallacious understanding of science.  First, no science deals with truth.  No science can deal with truth.  No scientist can ever know if what is "published" under the rubric of science has any relation to any truth.  Science deals with what seems to be revealed after repeated and partial observations of the world.  It's not methods or objectivity (by the way there is no such) that distinguishes sciences and scientists.  It's the desire and willingness to repeatedly observe the world from as many different standpoints as one can conceive that distinguishes scientific work.  That's it.  Peer review fits into science so conceived only in the sense that other scientists are willing to continue observing, bringing in new standpoints, collecting new information.  Second, the critiques of experimentation and the review of laboratory results presented in these papers is at best misplaced.  At worst wrongheaded.   What happens in laboratories, statistical testing, experiments has to be connected to things that don't happen in the laboratories, testing, etc. by creating a set of explanations, of stories if you will that encompass both.  Pasteur created in his laboratory ways to show the anthrax animal and its operations.  But his real genius was through the press, winning over other scientists, farmers, veterinarians, and convincing the local and national governments that the procedures he developed to kill this animal in the laboratory would also kill it on the actual farms and with actual farm animals; and that killing this animal would result in a reduction or elimination of the "awful disease" that was destroying European farms.  His assertion was sometimes wrong. But he was correct often enough that his process for killing the anthrax animal eventually was supported by the scientific community, by veterinarians, by public officials, and by farmers.  And over the ensuing years other approaches were developed based on continuing observations of this animal and others as well. This process is ongoing today, more than 100 years after Pasteur's death.

2.       Peer Review has been reduced to a review by fellow liars with a political agenda when it comes to "climate science" They all do it for the billions they get to "research" things that they will not publish with the appropriate supporting data. But don't expect this kind of reporting to make it to the Obama media harem couches at the NYTimes, WaPo, etc.

3.       The cover story of the Economist in Oct. of last year was about this very problem.  Here is a disturbing excerpt:

"A rule of thumb among biotechnology venture-capitalists is that half of published research cannot be replicated. Even that may be optimistic. Last year researchers at one biotech firm, Amgen, found they could reproduce just six of 53 "landmark" studies in cancer research. Earlier, a group at Bayer, a drug company, managed to repeat just a quarter of 67 similarly important papers. A leading computer scientist frets that three-quarters of papers in his subfield are bunk. In 2000-10 roughly 80,000 patients took part in clinical trials based on research that was later retracted because of mistakes or improprieties."

A link to the whole article:

http://www.economist.com/news/leaders/21588069-scientific-research-has-changed-world-now-it-needs-change-itself-how-science-goes-wrong

From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of David Wojick
Sent: Tuesday, July 15, 2014 1:03 PM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] Peer Review Scandals

There is a lot of junk in this article.

Here is the second paragraph: "Acoustics is an important field. But in biomedicine faulty research and a dubious peer-review process can have life-or-death consequences. In June, Dr. Francis Collins, director of the National Institutes of Health and responsible for $30 billion in annual government-funded research, held a meeting to discuss ways to ensure that more published scientific studies and results are accurate. According to a 2011 report in the monthly journal Nature Reviews Drug Discovery, the results of two-thirds of 67 key studies analyzed by Bayer researchers from 2008-2010 couldn't be reproduced."

Of course peer review has nothing to do with replication.

My guess is there are between 5 and 10 million peer reviews a year, but it only takes 4 or 5 anecdotes, some way off base, to generate broad claims of wholesale corruption, that is hurting science. This is what social movements feed on, and there is plenty to go around.

Interestingly, there is a metric angle to the JVC scandal. I think that with proper research an algorithm could be developed that will detect this sort of fraud. It would operate on the article submission tracking systems that all large publishers use. I discuss this in the comments to Kent Anderson's article here:
http://scholarlykitchen.sspnet.org/2014/07/14/trust-but-verify-identity-fraud-and-exploitation-of-the-trust-economy-in-scholarly-publishing/

David Wojick
http://insidepublicaccess.com/issues.html

At 09:14 AM 7/15/2014, you wrote:

So much for peer review.

Stephen J Bensman, Ph.D.
LSU Libraries
Lousiana State University
Baton Rouge, LA 70803
USA

WALL STREET JOURNAL  OPINION PIECE

The Corruption of Peer Review Is Harming Scientific Credibility
Dubious studies on the danger of hurricane names may be laughable. But bad science can cause bad policy.
By
Hank Campbell
July 13, 2014 6:32 p.m. ET
Academic publishing was rocked by the news on July 8 that a company called Sage Publications is retracting 60 papers from its Journal of Vibration and Control, about the science of acoustics. The company said a researcher in Taiwan and others had exploited peer review so that certain papers were sure to get a positive review for placement in the journal. In one case, a paper's author gave glowing reviews to his own work using phony names.
Acoustics is an important field. But in biomedicine faulty research and a dubious peer-review process can have life-or-death consequences. In June, Dr. Francis Collins, director of the National Institutes of Health and responsible for $30 billion in annual government-funded research, held a meeting to discuss ways to ensure that more published scientific studies and results are accurate. According to a 2011 report in the monthly journal Nature Reviews Drug Discovery, the results of two-thirds of 67 key studies analyzed by Bayer researchers from 2008-2010 couldn't be reproduced.
Enlarge Image Close
[http://si.wsj.net/public/resources/images/BN-DR115_edp071_D_201][cat]
Getty Images
That finding was a bombshell. Replication is a fundamental tenet of science, and the hallmark of peer review is that other researchers can look at data and methodology and determine the work's validity. Dr. Collins and co-author Dr. Lawrence Tabak highlighted the problem in a January 2014 article in Nature. "What hope is there that other scientists will be able to build on such work to further biomedical progress," if no one can check and replicate the research, they wrote.
The authors pointed to several reasons for flawed studies, including "poor training of researchers in experimental design," an "emphasis on making provocative statements," and publications that don't "report basic elements of experimental design." They also said that "some scientists reputedly use a 'secret sauce' to make their experiments workand withhold details from publication or describe them only vaguely to retain a competitive edge."
Papers with such problems or omissions would never see the light of day if sound peer-review practices were in placeand their absence at many journals is the root of the problem. Peer review involves an anonymous panel of objective experts critiquing a paper on its merits. Obviously, a panel should not contain anyone who agrees in advance to give the paper favorable attention and help it get published. Yet a variety of journals have allowed or overlooked such practices.
Absent rigorous peer review, we get the paper published in June in the Proceedings of the National Academy of Sciences. Titled "Female hurricanes are deadlier than male hurricanes," it concluded that hurricanes with female names cause more deaths than male-named hurricanesostensibly because implicit sexism makes people take the storms with a woman's name less seriously. The work was debunked once its methods were examined, but not before it got attention nationwide.
Such a dubious paper made its way into national media outlets because of the imprimatur of the prestigious National Academy of Sciences.
Yet a look at the organization's own submission guidelines makes clear that if you are a National Academy member today, you can edit a research paper that you wrote yourself and only have to answer a few questions before an editorial board; you can even arrange to be the official reviewer for people you know. The result of such laxity isn't just the publication of a dubious finding like the hurricane gender-bias claim. Some errors can have serious consequences if bad science leads to bad policy.
In 2002 and 2010, papers published in the Proceedings of the National Academy of Sciences claimed that a pesticide called atrazine was causing sex changes in frogs. As a result the Environmental Protection Agency set up special panels to re-examine the product's safety. Both papers had the same editor, David Wake of the University of California, Berkeley, who is a colleague of the papers' lead author, Tyrone Hayes, also of Berkeley.
In keeping with National Academy of Sciences policy, Prof. Hayes preselected Prof. Wake as his editor. Both studies were published without a review of the data used to reach the finding. No one has been able to reproduce the results of either paper, including the EPA, which did expensive, time-consuming reviews of the pesticide brought about by the published claims. As the agency investigated, it couldn't even use those papers about atrazine's alleged effects because the research they were based on didn't meet the criteria for legitimate scientific work. The authors refused to hand over data that led them to their claimed resultswhich meant no one could run the same computer program and match their results.
Earlier this month, Nature retracted two studies it had published in January in which researchers from the Riken Center for Development Biology in Japan asserted that they had found a way to turn some cells into embryonic stem cells by a simple stress process. The studies had passed peer review, the magazine said, despite flaws that included misrepresented information.
Fixing peer review won't be easy, although exposing its weaknesses is a good place to start. Michael Eisen, a biologist at UC Berkeley, is a co-founder of the Public Library of Science, one of the world's largest nonprofit science publishers. He told me in an email that, "We need to get away from the notion, proven wrong on a daily basis, that peer review of any kind at any journal means that a work of science is correct. What it means is that a few (1-4) people read it over and didn't see any major problems. That's a very low bar in even the best of circumstances."
But even the most rigorous peer review can be effective only if authors provide the data they used to reach their results, something that many still won't do and that few journals require for publication. Some publishers have begun to mandate open data. In March the Public Library of Science began requiring that study data be publicly available. That means anyone with the ability to check should be able to reproduce, validate and understand the findings in a published paper. This should also ensure that there is much better scrutiny of flawed claims about sexist weather events and hermaphroditic frogsbefore they appear on every news station in America.
Mr. Campbell is the founder of Science 2.0 and co-author of "Science Left Behind" (PublicAffairs, 2012).
Content-Type: image/jpeg;
         name="image001.jpg"
Content-Description: image001.jpg
Content-Disposition: inline;
         size=22479;
         creation-date=Tue, 15 Jul 2014 13:14:32 GMT;
         modification-date=Tue, 15 Jul 2014 13:14:32 GMT;
         filename="image001.jpg"
Content-ID: <image001.jpg at 01CFA004.CDBC0B40<mailto:image001.jpg at 01CFA004.CDBC0B40>>

Content-Type: image/jpeg;
         name="image002.jpg"
Content-Description: image002.jpg
Content-Disposition: inline;
         size=76790;
         creation-date=Tue, 15 Jul 2014 13:14:33 GMT;
         modification-date=Tue, 15 Jul 2014 13:14:33 GMT;
         filename="image002.jpg"
Content-ID: <image002.jpg at 01CFA004.CDBC0B40<mailto:image002.jpg at 01CFA004.CDBC0B40>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140715/28af8c6b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 22479 bytes
Desc: image001.jpg
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140715/28af8c6b/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 76790 bytes
Desc: image002.jpg
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140715/28af8c6b/attachment-0001.jpg>