The Chronicle, 18 November 2005

[Stephen J Bensman]

Since Loet has cited me, I should like to make some comments on
Monastersky's article and some of my findings on impact factor.

Monastersky interviewed me for that article, and, luckily for me, he did
not directly quote me on some of the things I had to say about impact
factor and academic evaluations.  In general, I found his article to be
unfair, since he concentrated on the shenanigans that are being played with
impact factor and supposed errors of ISI in constructing impact factor.
This makes for good copy but bad science.  Even the title of his article is
that of a grade B horror film like "The Blob That Ate New York."  There are
difficulties with impact factor, but these are not the result of editor's
shenanigans and ISI mistakes.  This would be obvious to anybody who has
correlated ISI citations with other measures of importance such as peer
ratings of academic programs.  I have obtained correlations in the 0.9
range, showing that ISI citations are virtually the equivalent of peer
ratings and have all the social biases of these ratings.  ISI citations are
not hard measures of quality but, in many respects, fuzzy outward
manifestations of a social stratification system operative in the human
mind.

In using any citation measure you should be absolutely clear as to your
goal in using them.  Journal importance is a multifaceted phenomenon, and
different citation measures will capture different facets of this
importance.  For example, I have always obtained much higher correlations
of ISI total citations with expert ratings of journals, because both these
measures are multidimensional and unlimited by elements of size and time.
Therefore, ISI total citations captures the historical importance of
journals, which is usually what scientists rate when asked to rank journals
by importance.  However, if I were to measure current importance, then I
probably would use some form of impact factor within strictly defined
journal types and disciplines so that, for example, chemistry research
journals would be compared to chemistry research journals and chemistry
review journals would be compared to chemistry review journals.  As another
example, if I were convinced that review articles are the epitome of
scientific writing, then I would use the standard impact factor.  One must
always keep in mind that there facets of journal importance that cannot be
captured by any citation measure but can be captured by other measures such
as library use, number of library holdings, or surveys of users.

In one respect Monastersky was right.  The use of impact factor for
purposes of academic evaluation is absolute madness as is any attempt to
construct a single citation measure that would capture precisely all facets
of journal importance.  No citation measure should be used unless its
characteristics have been explored by correlations with the results of
opinion surveys of those persons whom the citation measure concerns.  To do
otherwise is dictatorial.

I have ended my sermon for today.

Steve B.




Loet Leydesdorff <loet at LEYDESDORFF.NET>@LISTSERV.UTK.EDU> on 11/16/2005
12852:51 AM

Please respond to ASIS&T Special Interest Group on Metrics
       <SIGMETRICS at LISTSERV.UTK.EDU>

Sent by:    ASIS&T Special Interest Group on Metrics
       <SIGMETRICS at LISTSERV.UTK.EDU>


To:    SIGMETRICS at LISTSERV.UTK.EDU
cc:     (bcc: Stephen J Bensman/notsjb/LSU)

Subject:    [SIGMETRICS] The Chronicle, 18 November 2005

Journals maps and local impact factors<?xml:namespace prefix = o ns =
"urn:schemas-microsoft-com:office:office" />

    Comment on:
    Richard Monastersky, The Number That’s Devouring Science,
    Chronicle of Higher Education, October 14, 2005

Monasterky’s article lists a number of problems with the ISI-impact factor.
However, he fails to mention that the average impact factors vary among
fields of science. For example, impact factors in toxicology are
considerably lower than in immunology. This may be contributing to the
concern over the use of these types of measures.  Furthermore, Bensman
recently showed that more than with the impact factor faculty usage and
appreciation of journals correlates with the total citations given to a
journal. Citations can be considered a measure of a journal’s prestige,
while the impact factors follow the development of the fields at the
frontiers of research.
A fix to these problems might be a discipline-specific impact factor.
However, unambiguous clustering of the aggregated journal-journal citation
matrix into disciplines and specialties is impossible because the various
subsets overlap for very different reasons such as communalities in the
subject matter, methods, nationality, language, type of publisher or
purpose. Each journal has its own unique environment created in the acts of
citing and being-cited. Journals also differ in terms of their
within-journal (“self-citation”) rates.
These challenges recently led me to take a different tact.  Using ISI’s
Journal Citation Reports, I created the raw materials to make maps of the
citation neighborhoods of all the journals. The freeware program Pajek can
be used for the visualization. Clustering algorithms are available within
Pajek for differently colouring the visualizations; the input files are
available at http://users.fmg.uva.nl/lleydesdorff/jcr04 . The contributions
to the total number of citations in this local environment can be
considered as a local impact factor. This local impact can additionally be
corrected for within-journal citations. I used the horizontal axis of the
node for this corrected local impact, while the vertical axis is used for
the local impact including self-citations. All values are expressed as
percentages in order to control for differences in citation behaviour among
fields.
The advantages of this local impact factor are that (1) the normalization
on the total citations in the relevant citation environment is more
indicative of the intellectual status of a journal than an average
normalized over the number of publications like the impact factor of ISI;
(2) the evaluation can be made for each journal in the ISI-set and related
to the journal’s specific citation environment; (3) the correction for
within-journal citations is available both numerically and from the
visualizations. Furthermore, this information is freely available on the
internet.




Loet Leydesdorff
Amsterdam School of Communications Research (ASCoR)
Kloveniersburgwal 48, 1012 CX Amsterdam
Tel.: +31-20- 525 6598; fax: +31-20- 525 3681
loet at leydesdorff.net ; http://www.leydesdorff.net/

The Knowledge-Based Economy: Modeled, Measured, and Simulated
The Self-Organization of the Knowledge-Based Society; The Challenge of
Scientometrics