Peer Review Scandals

[David Wojick]

People do have a lot of strange ideas about science and Web 2.0 gives them 
a chance to indulge these ideas. The good news is that there is a lot of 
public engagement with science as a result, not all of it strange.

David

At 02:30 PM 7/15/2014, you wrote:
>Adminstrative info for SIGMETRICS (for example unsubscribe): 
>http://web.utk.edu/~gwhitney/sigmetrics.html
>Needless to say, it has provoked a lot of commentary­353 comments in one 
>day.  Three of them are below.  It is an op-ed piece in a newspaper with a 
>certain agenda, so do not expect too much.
>
>
>Stephen J Bensman, Ph.D.
>LSU Libraries
>Lousiana State University
>Baton Rouge, LA 70803
>
>1.       This entire discussion, including the articles in the NY Times, 
>Physics Today, and the Economist are based on a faulty, one might even say 
>fallacious understanding of science.  First, no science deals with 
>truth.  No science can deal with truth.  No scientist can ever know if 
>what is “published” under the rubric of science has any relation to any 
>truth.  Science deals with what seems to be revealed after repeated and 
>partial observations of the world.  It’s not methods or objectivity (by 
>the way there is no such) that distinguishes sciences and 
>scientists.  It’s the desire and willingness to repeatedly observe the 
>world from as many different standpoints as one can conceive that 
>distinguishes scientific work.  That’s it.  Peer review fits into science 
>so conceived only in the sense that other scientists are willing to 
>continue observing, bringing in new standpoints, collecting new 
>information.  Second, the critiques of experimentation and the review of 
>laboratory results presented in these papers is at best misplaced.  At 
>worst wrongheaded.   What happens in laboratories, statistical testing, 
>experiments has to be connected to things that don’t happen in the 
>laboratories, testing, etc. by creating a set of explanations, of stories 
>if you will that encompass both.  Pasteur created in his laboratory ways 
>to show the anthrax animal and its operations.  But his real genius was 
>through the press, winning over other scientists, farmers, veterinarians, 
>and convincing the local and national governments that the procedures he 
>developed to kill this animal in the laboratory would also kill it on the 
>actual farms and with actual farm animals; and that killing this animal 
>would result in a reduction or elimination of the “awful disease” that was 
>destroying European farms.  His assertion was sometimes wrong. But he was 
>correct often enough that his process for killing the anthrax animal 
>eventually was supported by the scientific community, by veterinarians, by 
>public officials, and by farmers.  And over the ensuing years other 
>approaches were developed based on continuing observations of this animal 
>and others as well. This process is ongoing today, more than 100 years 
>after Pasteur’s death.
>
>2.       Peer Review has been reduced to a review by fellow liars with a 
>political agenda when it comes to "climate science" They all do it for the 
>billions they get to "research" things that they will not publish with the 
>appropriate supporting data. But don't expect this kind of reporting to 
>make it to the Obama media harem couches at the NYTimes, WaPo, etc.
>
>
>3.       The cover story of the Economist in Oct. of last year was about 
>this very problem.  Here is a disturbing excerpt:
>
>"A rule of thumb among biotechnology venture-capitalists is that half of 
>published research cannot be replicated. Even that may be optimistic. Last 
>year researchers at one biotech firm, Amgen, found they could reproduce 
>just six of 53 “landmark” studies in cancer research. Earlier, a group at 
>Bayer, a drug company, managed to repeat just a quarter of 67 similarly 
>important papers. A leading computer scientist frets that three-quarters 
>of papers in his subfield are bunk. In 2000-10 roughly 80,000 patients 
>took part in clinical trials based on research that was later retracted 
>because of mistakes or improprieties."
>
>A link to the whole article:
>
>http://www.economist.com/news/leaders/21588069-scientific-research-has-changed-world-now-it-needs-change-itself-how-science-goes-wrong
>
>
>
>From: ASIS&T Special Interest Group on Metrics 
>[mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of David Wojick
>Sent: Tuesday, July 15, 2014 1:03 PM
>To: SIGMETRICS at LISTSERV.UTK.EDU
>Subject: Re: [SIGMETRICS] Peer Review Scandals
>
>Adminstrative info for SIGMETRICS (for example unsubscribe): 
><http://web.utk.edu/~gwhitney/sigmetrics.html>http://web.utk.edu/~gwhitney/sigmetrics.html 
>
>
>
>There is a lot of junk in this article.
>
>Here is the second paragraph: “Acoustics is an important field. But in 
>biomedicine faulty research and a dubious peer-review process can have 
>life-or-death consequences. In June, Dr. Francis Collins, director of the 
>National Institutes of Health and responsible for $30 billion in annual 
>government-funded research, held a meeting to discuss ways to ensure that 
>more published scientific studies and results are accurate. According to a 
>2011 report in the monthly journal Nature Reviews Drug Discovery, the 
>results of two-thirds of 67 key studies analyzed by Bayer researchers from 
>2008-2010 couldn't be reproduced.”
>
>Of course peer review has nothing to do with replication.
>
>My guess is there are between 5 and 10 million peer reviews a year, but it 
>only takes 4 or 5 anecdotes, some way off base, to generate broad claims 
>of wholesale corruption, that is hurting science. This is what social 
>movements feed on, and there is plenty to go around.
>
>Interestingly, there is a metric angle to the JVC scandal. I think that 
>with proper research an algorithm could be developed that will detect this 
>sort of fraud. It would operate on the article submission tracking systems 
>that all large publishers use. I discuss this in the comments to Kent 
>Anderson's article here:
><http://scholarlykitchen.sspnet.org/2014/07/14/trust-but-verify-identity-fraud-and-exploitation-of-the-trust-economy-in-scholarly-publishing/>http://scholarlykitchen.sspnet.org/2014/07/14/trust-but-verify-identity-fraud-and-exploitation-of-the-trust-economy-in-scholarly-publishing/ 
>
>
>David Wojick
><http://insidepublicaccess.com/issues.html>http://insidepublicaccess.com/issues.html
>
>
>At 09:14 AM 7/15/2014, you wrote:
>
>Adminstrative info for SIGMETRICS (for example unsubscribe): 
><http://web.utk.edu/~gwhitney/sigmetrics.html>http://web.utk.edu/~gwhitney/sigmetrics.html 
>
>So much for peer review.
>
>Stephen J Bensman, Ph.D.
>LSU Libraries
>Lousiana State University
>Baton Rouge, LA 70803
>USA
>
>
>WALL STREET JOURNAL  OPINION PIECE
>
>The Corruption of Peer Review Is Harming Scientific Credibility
>Dubious studies on the danger of hurricane names may be laughable. But bad 
>science can cause bad policy.
>By
>Hank Campbell
>July 13, 2014 6:32 p.m. ET
>Academic publishing was rocked by the news on July 8 that a company called 
>Sage Publications is retracting 60 papers from its Journal of Vibration 
>and Control, about the science of acoustics. The company said a researcher 
>in Taiwan and others had exploited peer review so that certain papers were 
>sure to get a positive review for placement in the journal. In one case, a 
>paper's author gave glowing reviews to his own work using phony names.
>Acoustics is an important field. But in biomedicine faulty research and a 
>dubious peer-review process can have life-or-death consequences. In June, 
>Dr. Francis Collins, director of the National Institutes of Health and 
>responsible for $30 billion in annual government-funded research, held a 
>meeting to discuss ways to ensure that more published scientific studies 
>and results are accurate. According to a 2011 report in the monthly 
>journal Nature Reviews Drug Discovery, the results of two-thirds of 67 key 
>studies analyzed by Bayer researchers from 2008-2010 couldn't be reproduced.
>Enlarge Image Close
>http://si.wsj.net/public/resources/images/BN-DR115_edp071_D_201
>cat
>
>Getty Images
>That finding was a bombshell. Replication is a fundamental tenet of 
>science, and the hallmark of peer review is that other researchers can 
>look at data and methodology and determine the work's validity. Dr. 
>Collins and co-author Dr. Lawrence Tabak highlighted the problem in a 
>January 2014 article in Nature. "What hope is there that other scientists 
>will be able to build on such work to further biomedical progress," if no 
>one can check and replicate the research, they wrote.
>The authors pointed to several reasons for flawed studies, including "poor 
>training of researchers in experimental design," an "emphasis on making 
>provocative statements," and publications that don't "report basic 
>elements of experimental design." They also said that "some scientists 
>reputedly use a 'secret sauce' to make their experiments work­and withhold 
>details from publication or describe them only vaguely to retain a 
>competitive edge."
>Papers with such problems or omissions would never see the light of day if 
>sound peer-review practices were in place­and their absence at many 
>journals is the root of the problem. Peer review involves an anonymous 
>panel of objective experts critiquing a paper on its merits. Obviously, a 
>panel should not contain anyone who agrees in advance to give the paper 
>favorable attention and help it get published. Yet a variety of journals 
>have allowed or overlooked such practices.
>Absent rigorous peer review, we get the paper published in June in the 
>Proceedings of the National Academy of Sciences. Titled "Female hurricanes 
>are deadlier than male hurricanes," it concluded that hurricanes with 
>female names cause more deaths than male-named hurricanes­ostensibly 
>because implicit sexism makes people take the storms with a woman's name 
>less seriously. The work was debunked once its methods were examined, but 
>not before it got attention nationwide.
>Such a dubious paper made its way into national media outlets because of 
>the imprimatur of the prestigious National Academy of Sciences.
>Yet a look at the organization's own submission guidelines makes clear 
>that if you are a National Academy member today, you can edit a research 
>paper that you wrote yourself and only have to answer a few questions 
>before an editorial board; you can even arrange to be the official 
>reviewer for people you know. The result of such laxity isn't just the 
>publication of a dubious finding like the hurricane gender-bias claim. 
>Some errors can have serious consequences if bad science leads to bad policy.
>In 2002 and 2010, papers published in the Proceedings of the National 
>Academy of Sciences claimed that a pesticide called atrazine was causing 
>sex changes in frogs. As a result the Environmental Protection Agency set 
>up special panels to re-examine the product's safety. Both papers had the 
>same editor, David Wake of the University of California, Berkeley, who is 
>a colleague of the papers' lead author, Tyrone Hayes, also of Berkeley.
>In keeping with National Academy of Sciences policy, Prof. Hayes 
>preselected Prof. Wake as his editor. Both studies were published without 
>a review of the data used to reach the finding. No one has been able to 
>reproduce the results of either paper, including the EPA, which did 
>expensive, time-consuming reviews of the pesticide brought about by the 
>published claims. As the agency investigated, it couldn't even use those 
>papers about atrazine's alleged effects because the research they were 
>based on didn't meet the criteria for legitimate scientific work. The 
>authors refused to hand over data that led them to their claimed 
>results­which meant no one could run the same computer program and match 
>their results.
>Earlier this month, Nature retracted two studies it had published in 
>January in which researchers from the Riken Center for Development Biology 
>in Japan asserted that they had found a way to turn some cells into 
>embryonic stem cells by a simple stress process. The studies had passed 
>peer review, the magazine said, despite flaws that included misrepresented 
>information.
>Fixing peer review won't be easy, although exposing its weaknesses is a 
>good place to start. Michael Eisen, a biologist at UC Berkeley, is a 
>co-founder of the Public Library of Science, one of the world's largest 
>nonprofit science publishers. He told me in an email that, "We need to get 
>away from the notion, proven wrong on a daily basis, that peer review of 
>any kind at any journal means that a work of science is correct. What it 
>means is that a few (1-4) people read it over and didn't see any major 
>problems. That's a very low bar in even the best of circumstances."
>But even the most rigorous peer review can be effective only if authors 
>provide the data they used to reach their results, something that many 
>still won't do and that few journals require for publication. Some 
>publishers have begun to mandate open data. In March the Public Library of 
>Science began requiring that study data be publicly available. That means 
>anyone with the ability to check should be able to reproduce, validate and 
>understand the findings in a published paper. This should also ensure that 
>there is much better scrutiny of flawed claims about sexist weather events 
>and hermaphroditic frogs­before they appear on every news station in America.
>Mr. Campbell is the founder of Science 2.0 and co-author of "Science Left 
>Behind" (PublicAffairs, 2012).
>Content-Type: image/jpeg;
>          name="image001.jpg"
>Content-Description: image001.jpg
>Content-Disposition: inline;
>          size=22479;
>          creation-date=Tue, 15 Jul 2014 13:14:32 GMT;
>          modification-date=Tue, 15 Jul 2014 13:14:32 GMT;
>          filename="image001.jpg"
>Content-ID: 
><<mailto:image001.jpg at 01CFA004.CDBC0B40>image001.jpg at 01CFA004.CDBC0B40>
>
>Content-Type: image/jpeg;
>          name="image002.jpg"
>Content-Description: image002.jpg
>Content-Disposition: inline;
>          size=76790;
>          creation-date=Tue, 15 Jul 2014 13:14:33 GMT;
>          modification-date=Tue, 15 Jul 2014 13:14:33 GMT;
>          filename="image002.jpg"
>Content-ID: 
><<mailto:image002.jpg at 01CFA004.CDBC0B40>image002.jpg at 01CFA004.CDBC0B40>
>
>Content-Type: image/jpeg;
>         name="image001.jpg"
>Content-Description: image001.jpg
>Content-Disposition: inline;
>         size=22479;
>         creation-date=Tue, 15 Jul 2014 18:30:33 GMT;
>         modification-date=Tue, 15 Jul 2014 18:30:33 GMT;
>         filename="image001.jpg"
>Content-ID: <image001.jpg at 01CFA02F.58FDD150>
>
>Content-Type: image/jpeg;
>         name="image002.jpg"
>Content-Description: image002.jpg
>Content-Disposition: inline;
>         size=76790;
>         creation-date=Tue, 15 Jul 2014 18:30:34 GMT;
>         modification-date=Tue, 15 Jul 2014 18:30:34 GMT;
>         filename="image002.jpg"
>Content-ID: <image002.jpg at 01CFA02F.58FDD150>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140715/13eb0b9b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 170dde7.jpg
Type: image/jpeg
Size: 22479 bytes
Desc: 170dde7.jpg
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140715/13eb0b9b/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 170ddf6.jpg
Type: image/jpeg
Size: 76790 bytes
Desc: 170ddf6.jpg
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140715/13eb0b9b/attachment-0001.jpg>