ART: Prathap, Arun's law of impact factor depreciation
Aparna Basu
aparna at NISTADS.RES.IN
Wed Jan 10 07:29:28 EST 2001
I would like to make a comment on the paper by Prathap, 'Arun's law..', and
would like to know what others on the list feel about it.
The possibility of evaluation using scientometric methods provokes people
into conducting the kind of exercise that Prathap has undertaken without
taking into account that scientometrics has a kind of statistical validity
which is eroded when very small specific samples are used. In that case,
refining techniques when the methodology itself is in question seems to be
missing the point.
In this connection I would also invite comments on using scientometric
evaluation for individual scientists/papers.
Aparna Basu
----- Original Message -----
From: Gretchen Whitney <gwhitney at UTK.EDU>
To: <SIGMETRICS at LISTSERV.UTK.EDU>
Sent: Tuesday, January 09, 2001 4:57 AM
Subject: [SIGMETRICS] ART: Prathap, Arun's law of impact factor depreciation
> Article reprinted with permission from Current Science.
>
> Gangan Prathap : e-mail : gp at cmmacs.ernet.in
>
> TITLE : Arun's law of impact factor depreciation
> AUTHOR Prathap G
> JOURNAL CURRENT SCIENCE 77: (11) 1405-1407 DEC 10 1999
>
> Document type: Article Language: English Cited References: 2
Times
> Cited: 0
>
> Addresses:
> Prathap G, Natl Aerosp Labs, Bangalore 560017, Karnataka, India.
> Natl Aerosp Labs, Bangalore 560017, Karnataka, India.
>
> Publisher:
> CURRENT SCIENCE ASSN, BANGALORE
>
>
> PAPER:
>
> Arun's law of impact factor depreciation
> CURRENT SCIENCE, VOL. 77, NO. 11, 10 December 1999
>
> In a very perceptive article recently, Arunachalam1 writes, 'Very often
> Indian authors publish papers in journals above a certain threshold impact
> factor (IF), but these papers do not get cited as often as would be
expected
> on the basis of the IFs of the journals. In an earlier paper I showed that
> Indian papers often lower the IF of journals'.
>
> Arunachalam goes on to quote the work of Tibor Braun and colleagues that
> 'in most fields the relative citation rates of India (ratios of actual
> citation rate/expected citation rate) is less than one'.
>
> When I related this observation to my wife, she drew my attention to an
> interesting sociological fact that several decades ago, as Asians (mainly
> from the Indian sub-continent) increasingly immigrated to UK, and
> inevitably, as Asian families moved into what were once all-white
> neighbourhoods, the real estate values of these areas fell immediately.
>
> So here, we have the bibliometric (scientometric) equivalent of a Law of
> Impact Factor Depreciation - an editor of an international journal accepts
> an Indian (or for that matter, any Third World) paper at great risk to the
> IF standing of her journal!
>
> Almost as if on cue, I was working on a very similar theme recently: the
> weakness of the IF criterion to assess the quality of scientific output of
> any state or institution.
>
> One way to approach the problem of Quantitative Research Assess-ment is
to
> employ the Impact Factor Approach. This is a prospective projection,
> assuming that a paper that has been published in a high IF journal is more
> likely to be well utilized or cited than a paper in a less prominent
> journal. We have considered Arunachalam's view that there is evidence to
the
> contrary. He writes in the same article, in lines immediately preceeding
the
> earlier quotation, 'One has to be extremely cautious in interpreting the
> data [IF] .... Ideally, one should count the number of times a paper is
> cited and see in which journals these citations occur, rather than merely
> look at the IF of the journal in which a paper is published'.
>
> My exercise, outlined below, showed that this is really true, by
carrying
> out a retrospective analysis. In fact, it will turn out that papers which
> have appeared in high IF journals may sink without a trace and that the
> really good papers will attract citations in excess of the number expected
> for that issue of that journal (the expected citation rate). What I have
> done is to track such papers which have a relative citation rate (RCR)
> greater than 1, implying that actual citations received by the paper are
> greater than that expected of a contribution in that category in that
> journal. To ensure that only the really good journals are screened, one
can
> consider only papers that have originally appeared in journals which have
an
> XCR greater than a prescribed and meaningful threshold.
> The exercise was carried out with some statistics I have of the National
> Aerospace Laboratories' (NAL) record in this aspect of research
assessment.
> Recently, we procured the Institute of Scientific Information's (ISI)
> Institutional Citation Report. Arguably, this allows us to compile an
> objective assessment of the published literature originating from NAL
during
> 1981-1997, the period covered by the ISI database.
>
> It is proposed to conduct the Research Assessment exercise by evaluating
the
> performance of published papers using the RCR > 1 criteria, where RCR is
> taken as the ratio of actual citations received to the expected citation
> rate (XCR). The XCR is the average citation per paper based on the journal
> title, year of publication and type of document. Thus, a paper published
> earlier should be expected to have more citations than the one published
> later. Categories (i.e. type of document) also matter. A full paper, a
> note and a letter to the editor may receive different citations. This is
why
> the RCR approach may be better than using the IF or citations approach as
is
> usually done.
>
> What I have done is to choose all items which have appeared from NAL
in
> a journal issue and category where XCR > 10 from the ISI database (57
papers
> out of the 587 papers that had NAL listed in one of the author's addresses
> in the ISI database). This is a strict criterion, considering that such
> distributions are highly skewed, with long tails, and with the mean likely
> to be very much to the right of the median. These papers have appeared in
> what can be considered to be the best journals ever used by NAL scientists
> during 1981-1997, implying that they have the highest IF. However, this
> does not mean that the paper which is fortunate to appear in such a
> prestigious journal will ever be used. In fact as Table 1 shows, the 57
> papers which belong to this category include many which have 0 and 1
> citations since they appeared! In fact more than half the papers in this
> list have RCR < 0.5, confirming Tibor Braun's assessment that the RCR of
> Indian papers is less than one.
>
> My further criterion is to select from this list of 57, only those
> papers which actually received citations in excess of XCR. This is again
an
> extremely strict criterion, especially considering the recent debate in
> Nature which establishes that papers from the Third World are often
> under-cited. Only 15 papers are found now (Table 2). Arguably, these are
> the best papers published from NAL during this period.
>
> The RCR criterion, more than the IF criterion, gives on a retrospective
> basis, an appreciation of what really are the papers that have been used
> over a well-defined period. Thus, this approach meets exactly
Arunachalam's
> prescription1 that 'one should count the number of times a paper is cited
> and see in which journals these citations occur, rather than merely look
at
> the IF of the journal in which a paper is published'.
>
> Discrimination here operates very unfairly at two levels. There is an
> accepted perception of discrimination regarding publication of papers. It
> is believed that a paper from 'weaker section' authors (e.g. women
> scientists, or those from the developing nations, as seen here) has to be
> much better than one from the 'stronger sections' to be accepted, i.e. the
> rejection criterion is more stringently applied to them. Seemingly, this
> would imply that their accepted publications would on an average, be of
> better quality. This is discrimination at one level.
>
> One would then expect that these papers would invite better citation
> rates. The operation of Arun's Law of IF depreciation is an expression of
> the fact that discrimination probably manifests at the citation level
too -
> that papers from the 'weaker sections', which may arguably be better than
> average, are fated to receive lower than average citations. Such concerns
> about region-based citation bias have appeared earlier2.
> 1. Arunachalam, S., Curr. Sci., 1999, 76, 1191-1203.
> 2. Paris, G., De Leo, G., Menozzi, P. and Gatto, M., Nature, 1998,
396,
> 210.
>
>
>
> GANGAN PRATHAP
> National Aerospace Laboratories,
> Bangalore 560 017, India
>
> Table 1. 57 'best' papers from NAL during 1981-1997 from the point of view
> of the quality of the journal as measured by expected citation rate above
a
> threshold value (XCR > 10) of the journal in which it appeared. Names
have
> been replaced by rank according to XCR to preserve anonymity
>
>
> Cites Expected Rank code Name of journal Year
> Type
> actually citation for name of
> received rate (XCR) first author
> 1.00 32.38 XCR-1 Phys. Rev.B 1987 N
> 1.00 24.17 XCR-2 J. Fluid Mech. 1981
> 4.00 21.93 XCR-3 J. Non-Cryst. 1984
> 35.00 21.07 XCR-4 J. Non-Cryst. 1981
> 22.00 21.07 XCR-5 J. Non-Cryst. 1981
> 5.00 20.95 XCR-6 Polymer 1983
> 6.00 18.93 XCR-7 IEEE Comput. 1983
> 25.00 18.70 XCR-8 J. Non-Cryst. 1983
> 0.00 18.39 XCR-9 Int. J. Fract. 1981
> 12.00 18.20 XCR-10 J. Non-Cryst. 1982
> 4.00 18.19 XCR-11 J. Appl. Phys. 1983
> 1.00 17.19 XCR-12 J. Phys. F 1981
> 51.00 16.73 XCR-13 J. Non-Cryst. 1986
> 16.00 16.69 XCR-14 Comput. Methods 1986
> 2.00 15.65 XCR-15 J. Appl. Phys. 1984
> 2.00 15.23 XCR-16 Phys. Rev. B 1991 N
> 5.00 15.13 XCR-17 J. Compos. Mater. 1984
> 3.00 14.76 XCR-18 J. Elec. Chem. 1986 N
> 0.00 14.51 XCR-19 Solid State Commun. 1987
> 0.00 14.25 XCR-20 J. Acoust. Soc. 1981
> 3.00 14.03 XCR-21 J. Phys. C 1982
> 1.00 13.66 XCR-22 Solid State Commun. 1982
> 4.00 13.62 XCR-23 J. Appl. Poly. 1981
> 10.00 13.48 XCR-24 J. Mater Sci. 1982
> 2.00 12.95 XCR-25 Solid State Commun. 1982
> 1.00 12.95 XCR-26 Solid State Commun. 1983
> 13.00 12.87 XCR-27 Int. J. Num. M. 1985
> 36.00 12.87 XCR-28 Int. J. Num. M. 1985
> 21.00 12.87 XCR-29 Int. J. Num. M. 1985
> 16.00 12.87 XCR-30 Int. J. Num. M. 1985
> 5.00 12.87 XCR-31 Int. J. Num. M. 1985
> 10.00 12.81 XCR-32 Int. J. Num. M. 1987
> 0.00 12.81 XCR-33 Int. J. Num. M. 1987
> 71.00 12.76 XCR-34 Int. J. Num. M. 1982
> 4.00 12.68 XCR-35 J. Mater. Sci. 1981
> 3.00 12.68 XCR-36 J. Mater. Sci. 1981
> 9.00 12.66 XCR-37 Talanta 1985
> 2.00 12.00 XCR-38 Solid State Commun. 1984
> 5.00 11.77 XCR-39 J. Appl. Phys. 1982 N
> 29.00 11.70 XCR-40 Int. J. Num. M. 1983
> 2.00 11.56 XCR-41 J. Mater. Sci. 1984
> 8.00 11.24 XCR-42 J. Mater. Sci. 1983
> 3.00 11.19 XCR-43 J. Appl. Poly. 1984
> 0.00 10.97 XCR-44 J. Elec. Chem. 1991
> 4.00 10.79 XCR-45 Atmos. Env. A 1990
> 2.00 10.65 XCR-46 Physica C 1992
> 18.00 10.43 XCR-47 Int. J. Num. M. 1986
> 6.00 10.43 XCR-48 Int. J. Num. M. 1986
> 26.00 10.43 XCR-49 Int. J. Num. M. 1986
> 15.00 10.43 XCR-50 Int. J. Num. M. 1986
> 12.00 10.43 XCR-51 Int. J. Num. M. 1986
> 0.00 10.42 XCR-52 Int. J. Num. M. 1984
> 13.00 10.30 XCR-53 Int. J. Num. M. 1988
> 2.00 10.17 XCR-54 J. Non-Cryst. 1988
> 9.00 10.17 XCR-55 J. Non-Cryst. 1988
> 6.00 10.12 XCR-56 Phys. Chem. Miner. 1991
> 9.00 10.11 XCR-57 J. Non-Cryst. 1990
>
> NAL papers arranged according to decreasing XCR which is the average
> citation per paper based on the journal title, year of publication and
type
> of document. Thus, a paper published earlier should be expected to have
> more citations than the one published later. Categories (i.e. type of
> document) also matter: A full paper, a note and a letter to the editor may
> receive different citations. I have chosen all items which have appeared
> from NAL in an issue and category where XCR>10. These can be considered
to
> be the best journals ever used by NAL scientists during this period
> (1981-1997), implying that they have the highest IF. It is easier to use
> this criterion than the IF values as the latter keep changing from year to
> year. However this does not mean that the paper which is fortunate to
> appear in such a prestigious journal will ever be used. In fact the 57
> papers which belong to this category include many which have 0 or 1
> citations since they appeared!
> Table 2. 15 papers from the 57 'best' papers of Table 1 which have RCR
>
> 1, i.e.
> actually received citations in excess of XCR, ranked according to RCR
>
> Relative Cites Expected XCR Name of Journal Year
> Citation actually citation rank
> Rate received rate (XCR from
> (RCR) Table1
> 5.56 71.00 12.76 XCR-34 Int. J. Num. M 1982
> 3.05 51.00 16.73 XCR-13 J. Non-Cryst. 1986
> 2.80 36.00 12.87 XCR-28 Int. J. Num. M. 1985
> 2.49 26.00 10.43 XCR-49 Int. J. Num. M. 1986
> 2.48 29.00 11.70 XCR-40 Int. J. Num. M. 1983
> 1.73 18.00 10.43 XCR-47 Int. J. Num. M. 1986
> 1.66 35.00 21.07 XCR-4 J. Non-Cryst. 1981
> 1.63 21.00 12.87 XCR-29 Int. J. Num. M. 1985
> 1.44 15.00 10.43 XCR-50 Int. J. Num. M. 1986
> 1.34 25.00 18.70 XCR-8 J. Non-Cryst. 1983
> 1.26 13.00 10.30 XCR-53 Int. J. Num. M. 1988
> 1.24 16.00 12.87 XCR30 Int. J. Num. M. 1985
> 1.15 12.00 10.43 XCR-51 Int. J. Num. M. 1986
> 1.04 22.00 21.07 XCR-5 J. Non-Cryst. 1981
> 1.01 13.00 12.87 XCR-27 Int. J. Num. M. 1985
>
> Performance of a published article has been evaluated using the RCR
> criterion. This is computed as the ratio of the actual citations received
by
> the item published to the expected citation rate, XCR. The criterion here
is
> to select from the list of 57 in Table 1, only those papers which actually
> received citations in excess of XCR (i.e. RCR >1). Only 15 papers are
found
> now. Arguably, these are the best papers published from NAL during this
> period. Note now the reversal of fortunes: The RCR rank has little
> correlation to the XCR rank. One more confounding factor when XCR or IF
> value is used to rank quality is that these values vary across
disciplines.
> Thus, in this instance, in a multi-disciplinary institutes like NAL,
> science-based papers earn much higher XCR than engineering-based papers
and
> a large share of the former appears in Table 1. The use of RCR removes
this
> complication, and the relative rankings have changed considerably.
>
> If one were to relax this criterion more generously, so that journals
which
> have XCR>5 are all included, then we find an enlarged number of 150 papers
> from NAL appearing in such a list. Under this relaxation, about 37 papers
> from NAL (out of 587 listed in the ISI database) have received RCR>1, i.e.
> citations in excess of the XCR=5 stipulation.
More information about the SIGMETRICS
mailing list