IMU Critique of Citation Analysis

Fri Jun 27 13:14:31 EDT 2008

Let me phrase what I wrote much more carefully.  In my opinion, the
accuracy of any rating measure is dependent on the level of consensus in
the field.  High levels of consensus lead to high levels of
inter-correlations among measures.  To show an example of this,
Mathematics was classified with the physical sciences in the 1993
National Research Council assessment of US research-doctorate programs,
and, in descending rank order, the correlations of the peer ratings of
these programs with citations per program faculty are below:

Chemistry - 0.81

Geosciences - 0.74

Astrophysics and Astronomy - 0.73

Physics - .070

Oceanography - 0.70

Computer Sciences - 0.61

Mathematics - 0.56

Statistics and Biostatistics - 0.17

The last seems to be some kind of outlier fluke and probably should be
discarded.  When that is done, Math is on the bottom and even below
Computer Sciences, which are related to engineering fields and therefore
not as academic as the other fields.  The low consensus in mathematics
as witnessed by the absence of dominant review journals probably plays a
role in Math's relatively low correlation.  With a correlation of only
0.56, I would be loath to rely on citations alone in any ratings of
mathematicians but utilize together with these other measures for
comparative and counterbalancing purposes.  However, the correlation is
high enough for citations to be informative.  

As for the distribution of journals by impact factor, there were 200
math journals covered by the 2007 SCI JCR, and these are all compressed
into a range of 2.739.  I would be careful of using impact factor as a
rating method for math journals, particularly below the top 10%.  

Stephen J. Bensman

LSU Libraries

Louisiana State University

Baton Rouge, LA   70803

USA

notsjb at lsu.edu

From: ASIS&T Special Interest Group on Metrics
[mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Eugene Garfield
Sent: Friday, June 27, 2008 11:09 AM
To: SIGMETRICS at listserv.utk.edu
Subject: Re: [SIGMETRICS] IMU Critique of Citation Analysis

I think you have to be careful about using the term citation analysis in
a blanket fashion. Even if the IMU was only concerned about rating
mathematicians there are no blanket statements that apply. Years ago I
did several citation analyses of mathematicians and math journals. I
have listed the group I find quickly from my web page:

Document Title

Journal Citation Studies. 36. Pure and Applied Mathematics Journals.
What they cite and vice versa. 
   http://www.garfield.library.upenn.edu/essays/v5p484y1981-82.pdf

<http://www.garfield.library.upenn.edu/essays/v5p484y1981-82.pdf#xml=htt
p://the-scientist.verity.library.upenn.edu/search97cgi/s97_cgi?action=Vi
ew&VdkVgwKey=http%3A%2F%2Fwww%2Egarfield%2Elibrary%2Eupenn%2Eedu%2Fessay
s%2Fv5p484y1981%2D82%2Epdf&doctype=xml&Collection=Essays&QueryZip=mathem
atics+or+math&> 

The 200 "Pure" Mathematicians Most Cited in 1978 and 1979, Including a
List of Most-Cited Publications for the Top 100 
   http://www.garfield.library.upenn.edu/essays/v5p666y1981-82.pdf

<http://www.garfield.library.upenn.edu/essays/v5p666y1981-82.pdf#xml=htt
p://the-scientist.verity.library.upenn.edu/search97cgi/s97_cgi?action=Vi
ew&VdkVgwKey=http%3A%2F%2Fwww%2Egarfield%2Elibrary%2Eupenn%2Eedu%2Fessay
s%2Fv5p666y1981%2D82%2Epdf&doctype=xml&Collection=Essays&QueryZip=mathem
atics+or+math&> 

The Multidisciplinary Impact of Math and Computer Science Is Reflected
in the 100 Most-Cited Articles in CompuMath Citation Index, 1976-1980 
   http://www.garfield.library.upenn.edu/essays/v7p232y1984.pdf

<http://www.garfield.library.upenn.edu/essays/v7p232y1984.pdf#xml=http:/
/the-scientist.verity.library.upenn.edu/search97cgi/s97_cgi?action=View&
VdkVgwKey=http%3A%2F%2Fwww%2Egarfield%2Elibrary%2Eupenn%2Eedu%2Fessays%2
Fv7p232y1984%2Epdf&doctype=xml&Collection=Essays&QueryZip=mathematics+or
+math&> 

The Most-Cited Physical-Sciences Publications in the 1945-1954 Science
Citation Index. Part 2. Twenty Citation Classics in Mathematics 
   http://www.garfield.library.upenn.edu/essays/v13p376y1990.pdf

<http://www.garfield.library.upenn.edu/essays/v13p376y1990.pdf#xml=http:
//the-scientist.verity.library.upenn.edu/search97cgi/s97_cgi?action=View
&VdkVgwKey=http%3A%2F%2Fwww%2Egarfield%2Elibrary%2Eupenn%2Eedu%2Fessays%
2Fv13p376y1990%2Epdf&doctype=xml&Collection=Essays&QueryZip=mathematics+
or+math&> 

gpd7213.tmp 

http://www.garfield.library.upenn.edu/essays/v15p439subjectindexy1992-93
.pdf

<http://www.garfield.library.upenn.edu/essays/v15p439subjectindexy1992-9
3.pdf#xml=http://the-scientist.verity.library.upenn.edu/search97cgi/s97_
cgi?action=View&VdkVgwKey=http%3A%2F%2Fwww%2Egarfield%2Elibrary%2Eupenn%
2Eedu%2Fessays%2Fv15p439subjectindexy1992%2D93%2Epdf&doctype=xml&Collect
ion=Essays&QueryZip=mathematics+or+math&> 

The Most-Cited Physical-Sciences Publications in the 1945-1954 Science
Citation Index. Part 2. Twenty Citation Classics in Mathematics 
   http://www.garfield.library.upenn.edu/essays/v13p381y1990.pdf

<http://www.garfield.library.upenn.edu/essays/v13p381y1990.pdf#xml=http:
//the-scientist.verity.library.upenn.edu/search97cgi/s97_cgi?action=View
&VdkVgwKey=http%3A%2F%2Fwww%2Egarfield%2Elibrary%2Eupenn%2Eedu%2Fessays%
2Fv13p381y1990%2Epdf&doctype=xml&Collection=Essays&QueryZip=mathematics+
or+math&> 

I never heard any objection from IMU or anyone else to these studies.
Also of interest is:

The 100 most-cited books in the CompuMath Citation Index, 1976-1980 
   http://www.garfield.library.upenn.edu/essays/v7p264y1984.pdf

<http://www.garfield.library.upenn.edu/essays/v7p264y1984.pdf#xml=http:/
/the-scientist.verity.library.upenn.edu/search97cgi/s97_cgi?action=View&
VdkVgwKey=http%3A%2F%2Fwww%2Egarfield%2Elibrary%2Eupenn%2Eedu%2Fessays%2
Fv7p264y1984%2Epdf&doctype=xml&Collection=Essays&QueryZip=fields+medal&>

Someone needs to do a citation analysis of the winners of Fields Medals
including Hi-Indexes. I haven't determined whether anyone has done
co-citation analyses of math subjects which would be a better way to
confirm your assertion that math is not susceptible to identifying
paradigm shifts. It all depends on how you do it and what you are
looking for. EG

________________________________

From: ASIS&T Special Interest Group on Metrics
[mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman
Sent: Friday, June 27, 2008 9:49 AM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: [SIGMETRICS] IMU Critique of Citation Analysis

In re the recent flap caused by the International Mathematical Union
critique of citation analysis, I checked the distribution of mathematics
journals by impact factor in the 2007 SCI JCR.  It was as I suspected.
The range of impact factors was only from 0.108 to
2.739--extraordinarily low and tight--and the top journals on the impact
factor had no review articles.  This is suggestive of an extremely
random citation pattern with no development of consensual paradigms.
Therefore, math acts like a humanities in terms of its literature use,
and citation analysis is probably not applicable to this discipline.  If
citation analysis is used, it has to be backed by other measures.

Stephen J. Bensman

LSU Libraries

Louisiana State University

Baton Rouge, LA   70803

USA

notsjb at lsu.edu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20080627/247d9115/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 88 bytes
Desc: image001.gif
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20080627/247d9115/attachment.gif>