Journal Prestige

[Quentin L. Burrell]

Thanks to Stephen for posting this article from the WSJ

I don't wish to say anything about the academic/commercial research aspect,
but it illustrates concerns that often arise particularly concerning the
(bio)medical literature. Let me pick out one quote:

"Top medical journals compete for papers describing large clinical trials
reporting small
               effects of treatments for diseases affecting many people,
although these reports often do not substantively
               advance scientific knowledge, and many subsequently are
invalidated."

A long standing worry for sceptical statisticians is the belief that results
of medical trials are only reported (published) if they are "statistically
significant" and then these are picked up by the media who conveniently
overlook the word "statistical".

For anyone interested in the more technical aspects of this sort of stuff,
there is an interesting article on "Why most published research findings are
false" at

http://medicine.plosjournals.org/archive/1549-1676/2/8/pdf/10.1371_journal.pmed.0020124-L.pdf

Again it is about medical research in the main.

(And it will please many list members in that it is OA - freely available
online!)

Seasonal greetings to list members!

Quentin

***************************************

Dr Quentin L Burrell
Isle of Man International Business School
The Nunnery
Old Castletown Road
Douglas
Isle of Man IM2 1QB
via United Kingdom
q.burrell at ibs.ac.im
www.ibs.ac.im



----- Original Message -----
From: "Stephen J Bensman" <notsjb at LSU.EDU>
To: <SIGMETRICS at LISTSERV.UTK.EDU>
Sent: Friday, December 30, 2005 7:07 PM
Subject: [SIGMETRICS] Journal Prestige








An interesting take on scientific journal prestige.  -- SB





                Mere Magazines


               By THOMAS P. STOSSEL
               December 30, 2005; Page A16


               Recently I was working in a Zambian orphanage when a young
woman with worsening shortness of breath and chest
               pain asked me for help. Armed only with a stethoscope, I
could do nothing other than diagnose a probable
               lethal tuberculous infection of the heart. Without devices
and drugs developed by companies, doctors are not
               very useful.


               It was therefore discouraging to return to my Boston-based
medical center and witness leading medical journals
               sanctimoniously demonizing not only the technologies
developed by drug companies but also the companies
               themselves. The Journal of the American Medical Association
has declared industry-sponsored research
               categorically untrustworthy, and, to publish it, demands that
an academic researcher be an author and take
               responsibility for its integrity, and also that an
independent academic statistician analyze its data. This
               and other journals rail obsessively against "financial
conflicts of interest" of academic researchers working
               with companies and conduct inquisitions to identify every
possible financial motive that might corrupt
               researchers' objectivity.


               The ongoing Merck situation is a case in point. The New
England Journal of Medicine wants the company to
               correct a five-year-old paper that, they allege,
inappropriately excluded three late-breaking adverse events
               associated with the painkiller Vioxx. The company has
correctly responded that published research projects
               always have defined beginnings and endings, and that it
reported all adverse events to the FDA. With the drug
               off the market and Merck mired in litigation, what problem
this correction would solve is unclear.
               Nevertheless, a Dec. 11 New York Times editorial excoriated
Merck for "manipulating a journal article" and
               informed doctors "that they will need to take the findings of
industry-backed studies with skeptical caution."


               The message in all this is clear: Medical academics are
saints -- devoted selflessly to patient care -- and
               corporate people are sinners, morally blinded by greed. But
having worked in academic medicine for over 35
               years and consulted for companies, this Manichean duality is
inconsistent with my experience and a woeful
               distortion of reality. In a Sept. 8 article in the New
England Journal of Medicine, I reported that no
               systematic evidence exists that corporate sponsorship of
academic research contributes to misconduct, bias,
               public mistrust or poor research quality.


               On the other hand, many academic colleagues working in my
field of basic biological research (I study how your
               body cells crawl around, which has no obvious commercial
value) would run over their grandmothers to claim
               priority for a discovery, impose their pet theory on the
field, obtain a research grant, win an award or
               garner a promotion. It's the same in other scientific fields,
and no wonder, because for relatively modest
               remuneration we compete for scarce resources and labor in
obscurity to achieve small advances few understand
               or appreciate. We exercise our ambitions by publishing
research papers in high-profile journals.


                                                                   * * *


               The research journal revolutionized scientific communication
in the 17th century. But until the scientific
               enterprise grew larger than the first journals could
accommodate, no peer review restricted publication. Once
               restrictions arose, human competitiveness established a
journal prestige pecking order that grew in importance
               as research became more prevalent and complex. The more
obscure one's research, the greater the premium on
               publishing it in a prestigious journal, where those who
administer limited rewards might see it, and where the
               news media are more likely to hype it.


               But unbeknownst to the media, the journals at the top got
there because of herd behavior by reqearchers, not
               because they are better than lower-tier journals at vetting
research quality. Here's why: Researchers submit
               their best work to the top journals, which can therefore
afford to maintain their prestige by rejecting, not
               publishing, many high quality papers. That's brand
creation -- not science. Most of their editorial effort
               goes into deciding which submitted papers are sufficiently
newsworthy. Anonymous peer review by jealous
               competitors has its merits, but it has a tendency to select
for fashionable if relatively unoriginal and
               inoffensive papers. Top medical journals compete for papers
describing large clinical trials reporting small
               effects of treatments for diseases affecting many people,
although these reports often do not substantively
               advance scientific knowledge, and many subsequently are
invalidated.


               And no description of medical research in a medical journal
comes close to the detail level or intense
               scrutiny imposed by the FDA on companies' documentation of
drug or device development before approval. Space
               constraints for readability and cost-savings preclude
journals from publishing detailed information on the
               order of what companies file with the FDA, and unpaid journal
peer reviewers, not to mention practicing
               doctors, would never read it anyway. The recent Korean
cloning fiasco, in which the leading science journals
               published blatantly fraudulent papers, wasn't the first such
incident to afflict prestige journals, and it
               could never happen under conditions of FDA review. Indeed,
doctors should take all studies published in
               "prominent medical journals" with "skeptical caution."


               The lower stringency of journals compared to the FDA is a
good thing, because academic biomedical research
               would come to a screeching halt if subjected to anything even
approximating FDA examination. Scientific
               knowledge advances reasonably efficiently, and new
technologies emerge, despite the looseness of journals. And
               researchers' craving for prestige goads them to greater
efforts.


               If reporters understood that journals are magazines, not Holy
Scripture, we might not be witnessing ever more
               onerous regulations inhibiting interactions between academic
and industry science. Prestigious biomedical
               journals are good for our health -- provided they stick to
their core business of facilitating imperfect
               communication between researchers. Leave drug and device
monitoring to the FDA -- and theology to theologians.


               Mr. Stossel is American Cancer Society Professor at Harvard
Medical School and co-director of the division of
               hematology at Brigham and Women's Hospital.