ART: Prathap, Arun's law of impact factor depreciation
Gretchen Whitney
gwhitney at UTK.EDU
Mon Jan 8 18:27:44 EST 2001
Article reprinted with permission from Current Science.
Gangan Prathap : e-mail : gp at cmmacs.ernet.in
TITLE : Arun's law of impact factor depreciation
AUTHOR Prathap G
JOURNAL CURRENT SCIENCE 77: (11) 1405-1407 DEC 10 1999
Document type: Article Language: English Cited References: 2 Times
Cited: 0
Addresses:
Prathap G, Natl Aerosp Labs, Bangalore 560017, Karnataka, India.
Natl Aerosp Labs, Bangalore 560017, Karnataka, India.
Publisher:
CURRENT SCIENCE ASSN, BANGALORE
PAPER:
Arun's law of impact factor depreciation
CURRENT SCIENCE, VOL. 77, NO. 11, 10 December 1999
In a very perceptive article recently, Arunachalam1 writes, 'Very often
Indian authors publish papers in journals above a certain threshold impact
factor (IF), but these papers do not get cited as often as would be expected
on the basis of the IFs of the journals. In an earlier paper I showed that
Indian papers often lower the IF of journals'.
Arunachalam goes on to quote the work of Tibor Braun and colleagues that
'in most fields the relative citation rates of India (ratios of actual
citation rate/expected citation rate) is less than one'.
When I related this observation to my wife, she drew my attention to an
interesting sociological fact that several decades ago, as Asians (mainly
from the Indian sub-continent) increasingly immigrated to UK, and
inevitably, as Asian families moved into what were once all-white
neighbourhoods, the real estate values of these areas fell immediately.
So here, we have the bibliometric (scientometric) equivalent of a Law of
Impact Factor Depreciation - an editor of an international journal accepts
an Indian (or for that matter, any Third World) paper at great risk to the
IF standing of her journal!
Almost as if on cue, I was working on a very similar theme recently: the
weakness of the IF criterion to assess the quality of scientific output of
any state or institution.
One way to approach the problem of Quantitative Research Assess-ment is to
employ the Impact Factor Approach. This is a prospective projection,
assuming that a paper that has been published in a high IF journal is more
likely to be well utilized or cited than a paper in a less prominent
journal. We have considered Arunachalam's view that there is evidence to the
contrary. He writes in the same article, in lines immediately preceeding the
earlier quotation, 'One has to be extremely cautious in interpreting the
data [IF] .... Ideally, one should count the number of times a paper is
cited and see in which journals these citations occur, rather than merely
look at the IF of the journal in which a paper is published'.
My exercise, outlined below, showed that this is really true, by carrying
out a retrospective analysis. In fact, it will turn out that papers which
have appeared in high IF journals may sink without a trace and that the
really good papers will attract citations in excess of the number expected
for that issue of that journal (the expected citation rate). What I have
done is to track such papers which have a relative citation rate (RCR)
greater than 1, implying that actual citations received by the paper are
greater than that expected of a contribution in that category in that
journal. To ensure that only the really good journals are screened, one can
consider only papers that have originally appeared in journals which have an
XCR greater than a prescribed and meaningful threshold.
The exercise was carried out with some statistics I have of the National
Aerospace Laboratories' (NAL) record in this aspect of research assessment.
Recently, we procured the Institute of Scientific Information's (ISI)
Institutional Citation Report. Arguably, this allows us to compile an
objective assessment of the published literature originating from NAL during
1981-1997, the period covered by the ISI database.
It is proposed to conduct the Research Assessment exercise by evaluating the
performance of published papers using the RCR > 1 criteria, where RCR is
taken as the ratio of actual citations received to the expected citation
rate (XCR). The XCR is the average citation per paper based on the journal
title, year of publication and type of document. Thus, a paper published
earlier should be expected to have more citations than the one published
later. Categories (i.e. type of document) also matter. A full paper, a
note and a letter to the editor may receive different citations. This is why
the RCR approach may be better than using the IF or citations approach as is
usually done.
What I have done is to choose all items which have appeared from NAL in
a journal issue and category where XCR > 10 from the ISI database (57 papers
out of the 587 papers that had NAL listed in one of the author's addresses
in the ISI database). This is a strict criterion, considering that such
distributions are highly skewed, with long tails, and with the mean likely
to be very much to the right of the median. These papers have appeared in
what can be considered to be the best journals ever used by NAL scientists
during 1981-1997, implying that they have the highest IF. However, this
does not mean that the paper which is fortunate to appear in such a
prestigious journal will ever be used. In fact as Table 1 shows, the 57
papers which belong to this category include many which have 0 and 1
citations since they appeared! In fact more than half the papers in this
list have RCR < 0.5, confirming Tibor Braun's assessment that the RCR of
Indian papers is less than one.
My further criterion is to select from this list of 57, only those
papers which actually received citations in excess of XCR. This is again an
extremely strict criterion, especially considering the recent debate in
Nature which establishes that papers from the Third World are often
under-cited. Only 15 papers are found now (Table 2). Arguably, these are
the best papers published from NAL during this period.
The RCR criterion, more than the IF criterion, gives on a retrospective
basis, an appreciation of what really are the papers that have been used
over a well-defined period. Thus, this approach meets exactly Arunachalam's
prescription1 that 'one should count the number of times a paper is cited
and see in which journals these citations occur, rather than merely look at
the IF of the journal in which a paper is published'.
Discrimination here operates very unfairly at two levels. There is an
accepted perception of discrimination regarding publication of papers. It
is believed that a paper from 'weaker section' authors (e.g. women
scientists, or those from the developing nations, as seen here) has to be
much better than one from the 'stronger sections' to be accepted, i.e. the
rejection criterion is more stringently applied to them. Seemingly, this
would imply that their accepted publications would on an average, be of
better quality. This is discrimination at one level.
One would then expect that these papers would invite better citation
rates. The operation of Arun's Law of IF depreciation is an expression of
the fact that discrimination probably manifests at the citation level too -
that papers from the 'weaker sections', which may arguably be better than
average, are fated to receive lower than average citations. Such concerns
about region-based citation bias have appeared earlier2.
1. Arunachalam, S., Curr. Sci., 1999, 76, 1191-1203.
2. Paris, G., De Leo, G., Menozzi, P. and Gatto, M., Nature, 1998, 396,
210.
GANGAN PRATHAP
National Aerospace Laboratories,
Bangalore 560 017, India
Table 1. 57 'best' papers from NAL during 1981-1997 from the point of view
of the quality of the journal as measured by expected citation rate above a
threshold value (XCR > 10) of the journal in which it appeared. Names have
been replaced by rank according to XCR to preserve anonymity
Cites Expected Rank code Name of journal Year
Type
actually citation for name of
received rate (XCR) first author
1.00 32.38 XCR-1 Phys. Rev.B 1987 N
1.00 24.17 XCR-2 J. Fluid Mech. 1981
4.00 21.93 XCR-3 J. Non-Cryst. 1984
35.00 21.07 XCR-4 J. Non-Cryst. 1981
22.00 21.07 XCR-5 J. Non-Cryst. 1981
5.00 20.95 XCR-6 Polymer 1983
6.00 18.93 XCR-7 IEEE Comput. 1983
25.00 18.70 XCR-8 J. Non-Cryst. 1983
0.00 18.39 XCR-9 Int. J. Fract. 1981
12.00 18.20 XCR-10 J. Non-Cryst. 1982
4.00 18.19 XCR-11 J. Appl. Phys. 1983
1.00 17.19 XCR-12 J. Phys. F 1981
51.00 16.73 XCR-13 J. Non-Cryst. 1986
16.00 16.69 XCR-14 Comput. Methods 1986
2.00 15.65 XCR-15 J. Appl. Phys. 1984
2.00 15.23 XCR-16 Phys. Rev. B 1991 N
5.00 15.13 XCR-17 J. Compos. Mater. 1984
3.00 14.76 XCR-18 J. Elec. Chem. 1986 N
0.00 14.51 XCR-19 Solid State Commun. 1987
0.00 14.25 XCR-20 J. Acoust. Soc. 1981
3.00 14.03 XCR-21 J. Phys. C 1982
1.00 13.66 XCR-22 Solid State Commun. 1982
4.00 13.62 XCR-23 J. Appl. Poly. 1981
10.00 13.48 XCR-24 J. Mater Sci. 1982
2.00 12.95 XCR-25 Solid State Commun. 1982
1.00 12.95 XCR-26 Solid State Commun. 1983
13.00 12.87 XCR-27 Int. J. Num. M. 1985
36.00 12.87 XCR-28 Int. J. Num. M. 1985
21.00 12.87 XCR-29 Int. J. Num. M. 1985
16.00 12.87 XCR-30 Int. J. Num. M. 1985
5.00 12.87 XCR-31 Int. J. Num. M. 1985
10.00 12.81 XCR-32 Int. J. Num. M. 1987
0.00 12.81 XCR-33 Int. J. Num. M. 1987
71.00 12.76 XCR-34 Int. J. Num. M. 1982
4.00 12.68 XCR-35 J. Mater. Sci. 1981
3.00 12.68 XCR-36 J. Mater. Sci. 1981
9.00 12.66 XCR-37 Talanta 1985
2.00 12.00 XCR-38 Solid State Commun. 1984
5.00 11.77 XCR-39 J. Appl. Phys. 1982 N
29.00 11.70 XCR-40 Int. J. Num. M. 1983
2.00 11.56 XCR-41 J. Mater. Sci. 1984
8.00 11.24 XCR-42 J. Mater. Sci. 1983
3.00 11.19 XCR-43 J. Appl. Poly. 1984
0.00 10.97 XCR-44 J. Elec. Chem. 1991
4.00 10.79 XCR-45 Atmos. Env. A 1990
2.00 10.65 XCR-46 Physica C 1992
18.00 10.43 XCR-47 Int. J. Num. M. 1986
6.00 10.43 XCR-48 Int. J. Num. M. 1986
26.00 10.43 XCR-49 Int. J. Num. M. 1986
15.00 10.43 XCR-50 Int. J. Num. M. 1986
12.00 10.43 XCR-51 Int. J. Num. M. 1986
0.00 10.42 XCR-52 Int. J. Num. M. 1984
13.00 10.30 XCR-53 Int. J. Num. M. 1988
2.00 10.17 XCR-54 J. Non-Cryst. 1988
9.00 10.17 XCR-55 J. Non-Cryst. 1988
6.00 10.12 XCR-56 Phys. Chem. Miner. 1991
9.00 10.11 XCR-57 J. Non-Cryst. 1990
NAL papers arranged according to decreasing XCR which is the average
citation per paper based on the journal title, year of publication and type
of document. Thus, a paper published earlier should be expected to have
more citations than the one published later. Categories (i.e. type of
document) also matter: A full paper, a note and a letter to the editor may
receive different citations. I have chosen all items which have appeared
from NAL in an issue and category where XCR>10. These can be considered to
be the best journals ever used by NAL scientists during this period
(1981-1997), implying that they have the highest IF. It is easier to use
this criterion than the IF values as the latter keep changing from year to
year. However this does not mean that the paper which is fortunate to
appear in such a prestigious journal will ever be used. In fact the 57
papers which belong to this category include many which have 0 or 1
citations since they appeared!
Table 2. 15 papers from the 57 'best' papers of Table 1 which have RCR >
1, i.e.
actually received citations in excess of XCR, ranked according to RCR
Relative Cites Expected XCR Name of Journal Year
Citation actually citation rank
Rate received rate (XCR from
(RCR) Table1
5.56 71.00 12.76 XCR-34 Int. J. Num. M 1982
3.05 51.00 16.73 XCR-13 J. Non-Cryst. 1986
2.80 36.00 12.87 XCR-28 Int. J. Num. M. 1985
2.49 26.00 10.43 XCR-49 Int. J. Num. M. 1986
2.48 29.00 11.70 XCR-40 Int. J. Num. M. 1983
1.73 18.00 10.43 XCR-47 Int. J. Num. M. 1986
1.66 35.00 21.07 XCR-4 J. Non-Cryst. 1981
1.63 21.00 12.87 XCR-29 Int. J. Num. M. 1985
1.44 15.00 10.43 XCR-50 Int. J. Num. M. 1986
1.34 25.00 18.70 XCR-8 J. Non-Cryst. 1983
1.26 13.00 10.30 XCR-53 Int. J. Num. M. 1988
1.24 16.00 12.87 XCR30 Int. J. Num. M. 1985
1.15 12.00 10.43 XCR-51 Int. J. Num. M. 1986
1.04 22.00 21.07 XCR-5 J. Non-Cryst. 1981
1.01 13.00 12.87 XCR-27 Int. J. Num. M. 1985
Performance of a published article has been evaluated using the RCR
criterion. This is computed as the ratio of the actual citations received by
the item published to the expected citation rate, XCR. The criterion here is
to select from the list of 57 in Table 1, only those papers which actually
received citations in excess of XCR (i.e. RCR >1). Only 15 papers are found
now. Arguably, these are the best papers published from NAL during this
period. Note now the reversal of fortunes: The RCR rank has little
correlation to the XCR rank. One more confounding factor when XCR or IF
value is used to rank quality is that these values vary across disciplines.
Thus, in this instance, in a multi-disciplinary institutes like NAL,
science-based papers earn much higher XCR than engineering-based papers and
a large share of the former appears in Table 1. The use of RCR removes this
complication, and the relative rankings have changed considerably.
If one were to relax this criterion more generously, so that journals which
have XCR>5 are all included, then we find an enlarged number of 150 papers
from NAL appearing in such a list. Under this relaxation, about 37 papers
from NAL (out of 587 listed in the ISI database) have received RCR>1, i.e.
citations in excess of the XCR=5 stipulation.
More information about the SIGMETRICS
mailing list