Impact Factor Shenanigans

[Stephen J Bensman]

Once again our informetric shenanigans have made headline news.  I do not consider this a particularly good treatment of the topic, and I am rather miffed that, although the reporter mentions the special issue of Scientometrics on the topic, he does not cite my article in it—just the article I trashed there.


Stephen J Bensman, Ph.D.
LSU Libraries
Lousiana State University
Baton Rouge, LA 70803
USA

[Description: The Wall Street Journal]
U.S. NEWS
August 24, 2012, 6:52 p.m. ET
Journals' Ranking System Roils Research
By GAUTAM NAIK<http://online.wsj.com/search/term.html?KEYWORDS=GAUTAM+NAIK&bylinesearch=true>
Growing pressure on scientific journals to increase their influence in the research world is pushing them to ever further lengths to play the system that ranks scholarly publications.
In July, a publication called Scientific World Journal retracted two papers about regenerative medicine, saying they had excessively cited another journal, Cell Transplantation.

[Description: image]
At issue was the "impact-factor ranking," one of the most influential numbers in scholarship. The impact factor was invented more than 50 years ago as a simple way to grade journals, on the basis of how frequently their articles got cited in the literature.
But concerns have arisen that some journals' impact factor is artificially inflated by excessive citations—which appears to be why the editors of The Scientific World Journal retracted previously published work.
"These articles have both been retracted on the basis that they violate The Scientific World Journal's policy against citation manipulation," the Scientific World Journal said in a statement on its web site.
In response, Thomson Reuters, which publishes the impact-factor numbers, suspended the rankings for both the Scientific World Journal and Cell Transplantation for two years, a blow to the researchers who publish in those journals.
The broader worry is that the once-obscure yardstick is now a ubiquitous tool for assessing scientific merit—a job it wasn't designed to do, and whose use is open to manipulation.
"These concerns are becoming a crescendo, as the number of papers has increased exponentially" in the last two decades, writes Jerome Vanclay, an ecologist at Southern Cross University in Australia, in a recent article in the journal Scientometrics.
The impact factor, or IF, is routinely used by researchers in deciding where to publish and what to read. It guides promotions, tenure decisions and funding committees around the world, who assume someone publishing in a high-impact journal must be doing superior work.
Thomson Reuters calculates the IF by dividing the number of citations of research papers in a journal in one year by the total number of papers published in the same journal in the two previous years. So while the IF captures the citation rate of a journal as a whole, it says nothing about the quality or veracity of any individual paper.
Nonetheless, more and more countries today use the IF system to grade scientists. A few years ago, Qatar University began offering cash bonuses to its academics linked to the IF of the journal in which they publish.
Critics say that pushes academics to seek trendy fields of research and to try to publish in journals with the highest IF, instead of those that offer the best audience for their work. "It distorts the entire scientific enterprise," says Fiona Godlee, editor of the British Medical Journal.
The IF is easily gamed, too. One in five academics in economics, sociology, psychology and business said they had been asked by editors to pad their papers with unnecessary citations to articles in the same journal, according to a study published in Science in February.
A year ago, Alberto Vincentelli, a professor of electrical engineering at the University of California at Berkeley, submitted a paper to a respected journal called IEEE Transactions on Circuits and Systems-I. The response surprised him.
In an email, the then editor of TCAS-I, Wouter Serdijn, said: "I don't think it is my duty to provide you with the exact references, but yours...If you feel that making such an effort is not necessary, then my recommendation is to withdraw the current manuscript from the TCAS-I review process."
The editor essentially "said I needed to cite more papers that had previously appeared in the same journal without indicating any such paper that would be relevant for the study," says Dr. Vincentelli. In other words, according to Dr. Vincentelli, the editor was trying to artificially boost the journal's impact factor.
Prof. Serdijn, a professor at Delft University of Technology, denies the claim. He says his suggestion was aimed at making the paper more relevant to the journal's readership. Dr. Vincentelli eventually agreed to cite the papers and his study appeared in January 2012.
Such occurrences have spurred a backlash. In 2010, for example, the Australian government said it would no longer use IFs in judging grant applications.
"The disapproval isn't about the metric itself but about its misuse," says Jim Testa, a vice president of editorial development and publisher relations at Thomson Reuters.
In April, Phil Davis, a publishing consultant who writes for a blog called The Scholarly Kitchen, noticed unusual citation patterns at Cell Transplantation.
In the blog, Mr. Davis noted that a review article published in another journal, Medical Science Monitor, had cited a total of 490 articles in the field, of which 445 were articles that had appeared in Cell Transplantation alone, in 2008 and 2009. Both those years were used to compute the 2010 impact factor for Cell Transplantation, and those citations apparently had an effect: the journal's IF rose from 5.126 in 2009 to 6.204 in 2010, a jump of 21%.
Mr. Davis notes three of the four editors of the Medical Science Monitor review article were also on Cell Transplantation's editorial board. Also, two MSM editors wrote a review in another journal, The Scientific World Journal, citing 124 papers. Of those, 96 were from Cell Transplantation in 2008 and 2009.
"Those two review articles alone were responsible for a more-than-50% increase in Cell Transplantation's impact factor for 2010," says Mr. Davis.
Mark Graczynski, executive publisher of Medical Science Monitor, says there was no effort between the journals to manipulate the IF. "It might just be coincidence that there's an overlap of some editors," he says.
Write to Gautam Naik at gautam.naik at wsj.com<mailto:gautam.naik at wsj.com>
A version of this article appeared August 25, 2012, on page A2 in the U.S. edition of The Wall Street Journal, with the headline: Grade Inflation Creeps Into Science Journals.
Copyright 2012 Dow Jones & Company, Inc. All Rights Reserved

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20120827/4932d409/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 2151 bytes
Desc: image001.gif
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20120827/4932d409/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 60390 bytes
Desc: image002.jpg
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20120827/4932d409/attachment.jpg>