Chronicle of Higher Education IF Article 2

[Stephen J Bensman]

The last sending was screwed up for some people, so I am trying a different
tack.  In case it fails there is a Word file containing it at the bottom.
-- SB

(Embedded image moved to file: pic13186.gif)The Chronicle of Higher
Education(Embedded image moved to file: pic08313.gif)Research

http://chronicle.com/free/v52/i08/08a01201.htm


                                                                            
                                      From the issue dated October 14, 2005 
                                                                            





The Number That's Devouring Science


The impact factor, once a simple way to rank scientific journals, has
become an unyielding yardstick for hiring, tenure, and grants


By RICHARD MONASTERSKY


In the beginning, during the late 1950s, it was just an innocent idea in
Eugene Garfield's head. A Philadelphia researcher who described himself as
a "documentation consultant," Mr. Garfield spent his free time thinking
about scientific literature and how to mine information from it.


He eventually dreamed up something he called an "impact factor,"
essentially a grading system for journals, that could help him pick out the
most important publications from the ranks of lesser titles. To identify
which journals mattered most to scientists, he proposed tallying up the
number of citations an average article in each journal received.


This accounting method sounds harmless enough. Outside academe, few people
have even heard of it. Mr. Garfield, though, now compares his brainchild to
nuclear energy: a force that can help society but can unleash mayhem when
it is misused.


Indeed, impact factors have assumed so much power, especially in the past
five years, that they are starting to control the scientific enterprise. In
Europe, Asia, and, increasingly, the United States, Mr. Garfield's tool can
play a crucial role in hiring, tenure decisions, and the awarding of
grants.


"The impact factor may be a pox upon the land because of the abuse of that
number," says Robert H. Austin, a professor of physics at Princeton
University.


Impact-factor fever is spreading, threatening to skew the course of
scientific research, say critics. Investigators are now more likely to
chase after fashionable topics — the kind that get into high-impact
journals — than to follow important avenues that may not be the flavor of
the year, says Yu-Li Wang, a professor of physiology at the University of
Massachusetts Medical School. "It influences a lot of people's research
direction."


That influence has also led to a creeping sense of cynicism about the
business of science publications. Journal editors have learned how to
manipulate the system, sometimes through legitimate editorial choices and
other times through deceptive practices that artificially inflate their own
rankings. Several ecology journals, for example, routinely ask authors to
add citations to previous articles from that same journal, a policy that
pushes up its impact factor. Authors who have received such requests say
that the practice veers toward extortion and represents a violation of
scientific ethics.


What's more, investigations into impact factors have revealed problems with
the basic data used by ISI, the company that tabulates citation statistics
and journals' impact factors. Started by Mr. Garfield in Philadelphia, ISI
was bought in 1992 by the Thomson Corporation, which has tried to transform
the citation enterprise into a more profitable operation by buying up
databases and promoting its products. With alarming frequency, editors are
finding fault with the impact factors that Thomson has issued.


"This was a serious concern," says Alan Nevill, a professor of
biostatistics at the University of Wolverhampton, in England, who took
issue with the calculations that ISI made regarding the Journal of Sports
Science, which he edits. "Academia is being held ransom by these
citations."


Far From Its Roots


It wasn't supposed to be this way. "We never predicted that people would
turn this into an evaluation tool for giving out grants and funding," says
Mr. Garfield.


Although he first mentioned the term "impact factor" in a publication in
1955, it wasn't until the 1960s that Mr. Garfield and a colleague fully
developed the concept to help them select the most important journals for a
new citation index, which has grown into one of the most widely used
citation tools in science and the social sciences. It didn't make sense,
they reasoned, to include only the journals that get the most citations,
because that would eliminate smaller publications. So they invented a type
of measurement that reflects the average number of citations per article
for each journal.


The basic definition has changed little since then, although the process of
calculating impact factors has become highly automated through the use of
computer algorithms, which trolled through 27 million citations last year.
In June, ISI issued its latest set of impact factors, for 5,968 science
journals and 1,712 social-science journals.


To calculate the most recent factor for the journal Nature, for example,
the company tallied the number of citations in 2004 to all of the articles
that Nature published in 2002 and 2003. Those citations were divided by the
number of articles the journal published in those two years, yielding an
impact factor of 32.182 — the ninth-highest of all journals. It is a number
that editors and publishers across the world lust after; more than half of
all science journals listed by ISI score below 1.


Impact factors caught on because they are an objective measure that serves
many purposes. Librarians can use them to decide which journals to purchase
and which to cancel. Editors and publishers can chart their journals'
impact factors to gauge their progress relative to competitors. And
scientists can examine the numbers to see where their research papers are
likely to get the most attention.


Higher-ranking journals, it turns out, do get a message out better. Matthew
B. Stanbrook, an assistant professor of medicine at the University of
Toronto, tracked what happened after 12 medical journals published a joint
statement on research authorship and sponsorship in 2001 — an unusual
situation that provided direct comparisons. Over the following 26 months,
the highest-impact journal received 100 times as many citations to the
article as the lowest one of the 12, Dr. Stanbrook reported at a conference
on peer review and publishing last month in Chicago. "There's a measurable
value associated with a high-impact journal, which indicates why those
journals are important," he says.


Over the years, impact factors have proved so attractive to scientists that
they started applying them not only to journals but also to researahers.
Ideally, evaluators would look at the number of citations an individual
paper receives or a scientist accumulates over his or her career — but that
process takes time and money. Impact factors provide a shortcut.


They also help in the modern world of ultraspecialized science. Members of
a tenure committee or a hiring panel find it increasingly difficult to
assess the papers of a candidate working outside their own subdiscipline,
so they use the impact factor of the journal in which the paper appeared as
a measure of the paper's quality. By that logic, evaluators rate a paper
more highly if it appears in a high-impact journal, regardless of what the
paper actually says.


Europeans cite another reason that impact factors are popular there. In
some countries, the community of researchers in a particular field is so
small that they all know each other and either collaborate or compete.
Using impact factors to assess individual scientists is seen as an
improvement over tapping into an old-boy network to make hiring and grant
decisions.


Fuzzy Math


But relying on impact factors to evaluate a person is statistically
dimwitted, say critics of its spreading influence. The measurement is just
an average of all the papers in a journal over a year; it doesn't apply to
any single paper, let alone to any author. For example, a quarter of the
articles in Nature last year drew 89 percent of the citations to that
journal, so a vast majority of the articles received far fewer than the
average of 32 citations reflected in the most recent impact factor.


Mr. Garfield and ISI routinely point out the problems of using impact
factors for individual papers or people. "That is something we have
wrestled with quite a bit here," says Jim Pringle, vice president for
development at Thomson Scientific, the division that oversees ISI. "It is a
fallacy to think you can say anything about the citation pattern of an
article from the citation pattern of a journal."


Such warnings have not helped. In several countries in Europe and Asia,
administrators openly use impact factors to evaluate researchers or
allocate money:
      In England, hiring panels routinely consider impact factors, says Mr.
      Nevill.
      According to Spanish law, researchers are rewarded for publishing in
      journals defined by ISI as prestigious, which in practice has meant
      in the upper third of the impact-factor listings.
      In China, scientists get cash bonuses for publishing in high-impact
      journals, and graduate students in physics at some universities must
      place at least two articles in journals with a combined impact factor
      of 4 to get their Ph.D.'s, says Martin Blume, editor in chief of the
      American Physical Society, who recently met with scientists in China.


The obsession with impact factors has also seeped into the United States,
although less openly. Martin Frank, executive director of the American
Physiological Society, says a young faculty member once told him about a
policy articulated by her department chair. She was informed that in order
to get tenure, scientists should publish in journals with an impact factor
above 5.


"We are slaves to the impact factor," says Mr. Frank, whose organization
publishes 14 science journals.


Impact ranking may now be a tool that controls scientists, rather than the
other way around. Pressure to publish in the highest-impact science
journals — Nature, Science, and Cell — has led researchers to compete more
and more for the limited number of slots in those broader journals, thus
diminishing the specialty titles that have traditionally served as the main
publications of each discipline. Academe used to be a "publish or perish"
world, but now the halls of science have turned into a "publish in a
high-impact journal or perish" environment, says Massachusetts' Mr. Wang.


He observes that impact factors may even be affecting what kind of research
is conducted. Top journals require that papers be topical, in addition to
presenting important science, so researchers are shifting the kinds of
questions they investigate to accommodate those high-impact journals. "The
system is going after the short term," says Mr. Wang.


"For example, it is easy to catch attention when one describes a previously
unknown gene or protein related to a disease, even if the analysis is done
only superficially," he says. "Follow-up studies, to uncover the true
functions of the molecules or sometimes to challenge the initial analysis,
are typically more difficult to publish in journals of top 'impact.'"


Catherine D. DeAngelis, editor of the high-impact Journal of the American
Medical Association, also criticizes the current culture. The impact factor
"has taken on a life of its own," she says, lamenting that many scientists
view their work as a failure if they can't get into a top journal. "There
are wonderful journals that have impact factors lower than some of the
higher-citation journals, and they're perfectly appropriate for good
scientists to publish in."


The whole system has led to increasing discontent among researchers, says
Dr. DeAngelis. "It's bad for science in that you don't make researchers
feel good about what they're doing and the fact that their work gets
published in a good journal," she says. "That's bad. You're a better
scientist if you're a happy scientist."


Researchers go to great lengths to place their papers in high-impact
journals. They will often flip a manuscript from one publication to the
next, dropping reluctantly down the impact ladder until they find one that
will accept their work. The system slows the pace of science, say critics,
because researchers spend their time trying to publish their work rather
than moving on to the next set of experiments.


Sometimes authors will put considerable extra work into a paper — at the
request of reviewers at top journals — only to find it eventually rejected.
"I'll get so exhausted `y the whole thing that I won't even publish it or
will delay it for a year," says Princeton's Mr. Austin.


Think Quick


Deluged by so many manuscripts, high-impact journals can send only a
fraction out to experts for review. Nature, for example, rejects half of
the submissions it gets without forwarding them to referees, says its
editor in chief, Philip Campbell.


Mr. Austin worries about that process, saying that journal editors are
summarily rejecting unfashionable papers. "That can really limit
creativity, and really pioneering papers will not necessarily be judged as
such by these editors," he says, adding that the editors at top journals
are not active researchers.


Mr. Campbell responds that editors at Nature all have research experience
at good labs and keep on top of their fields by attending conferences and
reviewing the literature. "They are better than moqt academics in keeping
track of what's going on," he says. "I would put them up against any
academic any day in terms of knowing what's going on."


He also rejectq a belief widely held among scientists that Nature rejects
manuscripts if editors suspect that they won't attract citations and
therefore will depress the journal's impact factor. If that were true, he
says, the journal would stop publishing papers in geology or paleontology,
which rarely receive as many citations as ones in molecular biology.


"We're perfectly happy with the fact that we publish papers that are much
less cited than others," says Mr. Campbell, who also notes that Nature has
regularly voiced skepticism about impact factors in editorials, letters,
and news articles.


Many other editors contacted by The Chronicle also deny making judgments on
the basis of whether a paper will attract citations. But Dr. DeAngelis, of
JAMA, says editors at some top journals have told her that they do consider
citations when judging some papers. "There are people who won't publish
articles," she says, "because it won't help their impact factor."


She acknowledges that citations sometimes play a role in her own decisions
about a paper. "If I'm on the edge and we're going back and forth," she
says, "I might make the decision saying, Will people use this? In that
case, one of the criteria is: Will they cite it?"


Yet she also publishes papers that she knows will hurt JAMA's impact
factor. "We have a special theme issue on medical education, and we
continue to do it," she says, even though articles in it are cited
relatively infrequently.


Fiona Godlee, editor of BMJ (formerly known as the British Medical Journal
), agrees that editors take impact factors into account when deciding on
manuscripts, whether they realize it or not. "It would be hard to imagine
that editors don't do that," she says. "That's part of the way that impact
factors are subverting the scientific process."


She says editors may be rejecting not only studies in smaller or
less-fashionable fields, but also important papers from certain regions of
the world, out of fear that such reports won't attract sufficient citation
attention. "It's distorting people's priorities," she says, "and we have to
constantly fight against that."


Cult of the Factor


Although impact factors have been around for decades, it is only within the
past 10 years that they have taken on cult status, as the growing use of
the Internet has given researchers easy access to ISI data. The company
says the ranking is here to stay.


"One thing we won't do is change the impact factor as it stands now, just
because it's become such a key indicator over time," says Mr. Pringle, the
vice president for development. Rather than alter the original, ISI has
added additional information and measurement tools to complement the impact
factor, he says.


But the number continues to be so influential that some who run journals
try to manipulate the system. "Publishers have become quite expert in
skewing it to their own benefit," says Vitek Tracz, chairman of Current
Science Group, which publishes more than 100 open-access journals.


One well-known method is to publish more review articles — those that give
overviews of a topic but don't usually present new data. They generally
attract more citations than do original research articles. So when the
editorial board of the Journal of Environmental Quality met in 2003, it
resolved to emphasize review articles in order to shore up the journal's
slipping impact factor.


Other tactics exploit gaps in the way ISI calculates the impact factor.
When journals publish news articles, editorials, book reviews, and
abstracts of meetings, ISI does not count those items as "citable
articles"; hence they do not go into the denominator of the impact-factor
calculation. But if those uncounted items get cited in the literature, ISI
still puts those citations into the numerator, thereby increasing the
journal's impact factor.


Managers at ISI and several journal editors contacted by The Chronicle
dismissed the issue, arguing that news articles and editorials do not get
cited often. On average that may be true. But some of them gain enough
citations to significantly boost the impact factors of certain journals,
says Henk F. Moed, a bibliometrician at the Center for Science and
Technology Studies at Leiden University, in the Netherlands, who wrote
about the issue in his new book, Citation Analysis in Research Evaluation
(Springer, 2005). His analysis of the high-impact journal The Lancet, for
example, showed that free citations from news articles and similar material
buoyed the British medical journal's impact factor by 16 percent in 2002.


Many journals have added a considerable number of uncountable items to
their mix in recent years, even as they have decreased the number of
original research articles. In fact, Cell, JAMA, The Lancet, Nature, The
New England Journal of Medicine, and Science are all now publishing fewer
countable research items than they were in 1998, according to ISI data.


At the same time, those top journals and others have made a science out of
getting publicity for their products. Big journals with well-funded
public-relations offices send alerts to hundreds of reporters each week
about the articles slated for their next issues. The system generates news
items, which have been shown to increaqe citations to the original
scientific articles, thus raising impact factors. Smaller, less-visible
journals don't benefit from the same media connection.


Crooked Citations


Editors defend the changes they have made in their journals, arguing that
editorials, book reviews, news sections, and similar features are important
and popular with readers. But journal watchers point to other, less
scrupulous, ways to raise the citation numbers.


Sometimes journals will run editorials that cite numerous articles from
previous issues. In a new study, Jan Reedijk, of Leiden University, and Mr.
Moed found that a significant number of journals get a noticeable jump in
their impact factors from such self-citations in editorials.


In other cases, research articles in a journal preferentially cite that
very journal, with the effect of raising its impact factor. ISI detected a
clear example of that practice at the World Journal of Gastroenterology.
The company stopped listing that journal this year because 85 percent of
the citations to the publication were coming from its own pages. (Despite
that censure, the journal's Web site has a moving banner that still
trumpets its 2003 impact factor.)


The gaming has grown so intense that some journal editors are violating
ethical standards to draw more citations to their publications, say
scientists. John M. Drake, a postdoctoral researcher at the National Center
for Ecological Analysis and Synthesis, at the University of California at
Santa Barbara, sent a manuscript to the Journal of Applied Ecology and
received this e-mail response from an editor: "I should like you to look at
some recent issues of the Journal of Applied Ecology and add citations to
any relevant papers you might find. This helps our authors by drawing
attention to their work, and also adds internal integrity to the Journal's
themes."


Because the manuscript had not yet been accepted, the request borders on
extortion, Mr. Drake says, even if it weren't meant that way. Authors may
feel that they have to comply in order to get their papers published.
"That's an abuse of editorial power," he says, "because of the apparent
potential for extortion."


Robert P. Freckleton, a research fellow at the University of Oxford who is
the journal editor who sent the message to Mr. Drake, says he never
intended the request to be read as a requirement. "I'd be upset if people
read it that way," he says. "That's kind of a generic line we use. We
understand most authors don't actually do that." He changed the wording in
the form letter last week to clear up misunderstandings, he said.


Whatever the intention behind such requests, they are becoming more common.
Anurag A. Agrawal, an assistant professor of ecology and evolutionary
biology at Cornell University, has documented similar practices at five
other ecology journals. "It's embarrassing, and it's a scar on our
discipline," he says. "Authors are being asked to compromise their
principles. That chips away at the fabric of the scientific enterprise."


Mr. Freckleton defends the practice: "Part of our job as editors is making
sure that our work is getting cited and read appropriately." The policy, he
says, is not an explicit attempt to raise the journal's impact factor.


But the policy has done just that, and quite successfully, according to the
The Chronicle's analysis of self-citations to one-year-old articles — which
are important in the impact calculation. In 1997 the Journal of Applied
Ecology cited its own one-year-old articles 30 times. By 2004 that number
had grown to 91 citations, a 200-percent increase. Similar types of
citations of the journal in other publications had increased by only 41
percent.


The journal was engaged in other questionable activities at the time. Steve
Ormerod, executive editor from 2000 through 2004, wrote several editorials
during his tenure that cited his own journal dozens of times. In 2002, for
example, two of his commentaries cited 103 papers published in the journal
during 2000 and 2001. Those two editorials alone raised his journal's 2002
impact factor by 20 percent.


Mr. Ormerod, a professor of ecology at Cardiff University, in Wales,
acknowledges that his actions look suspicious, but says "there is a
less-sinister explanation." He was attempting, he says, to make the journal
more relevant by examining whether past articles on environmental issues
had led to policy actions. "As an accident, the impact factor went up at
the same time as self-citations went up," he says. He advocates removing
self-citations from the impact calculations completely, to avoid any
semblance of impropriety.


Nonetheless, the self-citations at his publication had a measurable effect.
The ecology journal's impact factor jumped from 1.3 in 1997 to 3.3 in 2004,
and its ranking within the discipline rose from 29th out of 86 journals to
16th out of 107.


Following inquiries by The Chronicle, Mr. Freckleton said last week he was
developing a plan to alter the journal's editorials so that self-citations
will not raise its impact factor.


Complaints From Researchers


ISI says it is taking steps to stay ahead of the schemers. "It's not easy,
but as we become aware of possible abuse, we try to expose that," says
Marie E. McVeigh, product-development manager. For example, citation
reports now indicate what percentage of citations to a journal come from
that same publication.


While it is trying to track abuse from editors, however, ISI may not be
doing enough to police itself. Several editors contacted by The Chronicle
have raised complaints about errors in the company's data and analyses. The
problems appear to be growing worse.


Mr. Blume, of the American Physical Society, says researchers have
contacted him recently to complain that the ISI database is missing
citations to their articles. "Complaints are on the rise," says Mr. Blume,
whose organization is looking into the concerns.


Mr. Nevill, editor in chief of the Journal of Sports Science, says his
journal suffered when ISI incorrectly counted short meeting abstracts as if
they were full-fledged original research articles or reviews. That
miscoding doubled the number of articles credited to the journal each year,
halving its impact factor, he says.


Dr. Godlee, of BMJ, says ISI incorrectly counted some items in her journal,
such as commentaries, with the effect of depressing its impact factor.


James Testa, director of editorial development at Thomson Scientific, takes
issue with calling those cases "errors." Questions often arise about how to
define certain types of articles, and ISI works closely with publishers to
establish a correct coding system for each journal, he says. The company
has decided to rerun its impact-factor calculations this year to correct
problems with 10 to 15 journals, says Mr. Pringle, of Thomson Scientific.
He says the rising importance of impact factors in science has caused
editors to pay closer attention to the calculations, which results in them
raising more complaints than in the past.


Like many other editors and researchers, Dr. Godlee sees an easy solution
to the types of problems that have been plaguing the calculations, as well
as to deliberate deceptions. She suggests that ISI count citations only to
original research articles, eliminating the problem of news stories,
editorials, reviews, and other kinds of materials. But ISI has steadfastly
resisted altering its original formula.


Given the power of ISI and its impact factors, scientists have little
choice but to accept the system — although competitors are emerging that
could alter the situation. And the growing use of online journals and
open-access journals could eventually topple the traditional system of
packaging articles into issues of a journal.


Like music lovers who download single songs instead of buying complete
albums, some researchers are starting to download only the articles they
want, regardless of where they originally appeared. "In terms of where it
gets published, it's becoming less and less an issue," says Harold P.
Erickson, a professor of cell biology at Duke University.


But most scientists still see value in differentiating between the quality
of articles in, say, Science and Science of the Total Environment. Even Mr.
Erickson has to face a dean who expects his professors to demonstrate their
excellence by occasionally publishing in Cell, Nature, Science, and other
journals with soaring impact factors.


http://chronicle.com
Section: Research & Publishing
Volume 52, Issue 8, Page A12
           Copyright © 2005 by The Chronicle of Higher Education

   Subscribe | About The Chronicle | Contact us | Terms of use | Privacy
                               policy | Help
                      (See attached file: CHE IF.doc)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pic13186.gif
Type: image/gif
Size: 1218 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20051010/7f2b1572/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pic08313.gif
Type: image/gif
Size: 3746 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20051010/7f2b1572/attachment-0001.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: CHE IF.doc
Type: application/msword
Size: 103424 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20051010/7f2b1572/attachment.doc>