PS Citation statistics

Mon Jun 16 10:00:51 EDT 2008

For the hell of it, I just checked the correlation between peer ratings
and citation per faculty member in the evaluation of US
research-doctorate programs conducted by the National Research Council
in 1993.  It was a mere 0.56--not high enough to solicit much
confidence.  The corresponding correlations for chemistry and physics
were 0.81 and 0.70.  These correlations would rise significantly if
total cites per program were substituted, but it does indicate that math
is somewhat of a different kettle of fish.

Stephen J. Bensman
LSU Libraries
Louisiana State University
Baton Rouge, LA   70803
USA
notsjb at lsu.edu
-----Original Message-----
From: Stephen J Bensman 
Sent: Monday, June 16, 2008 8:49 AM
To: 'ASIS&T Special Interest Group on Metrics'
Subject: RE: [SIGMETRICS] Citation statistics

In re the discussion here.  Mathematics be a peculiar field in that it
acts more like a humanities than a science.  The literature cited is
much older, and cites and library use are distributed much more
randomly.  Moreover, I have a funny feeling that the impact factor
distribution may be Poisson or binomial due to the absence of dominant
review journals due to an inability to form consensual paradigms.  I
discussed this matter with the chairman of the LSU math department, and
he stated that mathematicians not only do not know the answers, they do
not even know the questions.  He also pointed out that it is always
"Mathematics and the Sciences" in group classifications, indicating that
math is something different.  It is like "Social and the Behavioral
Sciences."  If this is the case, and math acts more like a humanities
than a science, then citation analysis may be out of the question, and
it is necessary to rely on peer judgment.

Stephen J. Bensman
LSU Libraries
Louisiana State University
Baton Rouge, LA   70803
USA
notsjb at lsu.edu

-----Original Message-----
From: ASIS&T Special Interest Group on Metrics
[mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stevan Harnad
Sent: Monday, June 16, 2008 8:20 AM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] Citation statistics

On Sun, 15 Jun 2008, Loet Leydesdorff wrote:

>> SH: But what all this valuable, valid cautionary discussion overlooks
is not
>> only the possibility but the *empirically demonstrated fact* that
there
>> exist metrics that are highly correlated with human expert rankings.
>
> It seems to me that it is difficult to generalize from one setting in
which
> human experts and certain ranks coincided to the *existence *of such
> correlations across the board. Much may depend on how the experts are
> selected. I did some research in which referee reports did not
correlate
> with citation and publication measures.

Much may depend on how the experts are selected, but that was just as
true during the 20 years in which rankings by experts were the sole
criterion for the rankings in the UR Research Assessment Exercise
(RAE). (In validating predictive metrics one must not endeavor to be
Holier than the Pope: Your predictor can at best hope to be as good as,
but not better than, your criterion.)

That said: All correlations to date between total departmental author
citation counts (not journal impact factors!) and RAE peer rankings
have been positive, sizable, and statistically significant for the
RAE, in all disciplines and all years tested. Variance there will be,
always, but a good-sized component from citations alone seems to be
well-established. Please see the studies of Professor Oppenheim and
others, for example as cited in:

    Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated
online
    RAE CVs Linked to University Eprint Archives: Improving the UK
Research
    Assessment Exercise whilst making it cheaper and easier. Ariadne 35.
    http://www.ariadne.ac.uk/issue35/harnad/

> Human experts are necessarily selected from a population of experts,
and it
> is often difficult to delineate between fields of expertise.

Correct. And the RAE rankings are done separately, discipline by
discipline; the validation of the metrics should be done that way too.

Perhaps there is sometimes a case for separate rankings even at
sub-disciplinary level. I expect the departments will be able to sort
that out. (And note that the RAE correlations do not constitute a
validation of metrics for evaluating individuals: I am confident that
that too will be possible, but it will require many more metrics and
much more validation.)

> Similarly, we
> know from quite some research that citation and publication practices
are
> field-specific and that fields are not so easy to delineate. Results
may be
> very sensitive to choices made, for example, in terms of citation
windows.

As noted, some of the variance in peer judgments will depend on the
sample of peers chosen; that is unavoidable. That is also why "light
touch" peer re-validation, spot-checks, updates and optimizations on the
initialized metric weights are also a good idea, across the years.

As to the need to evaluate sub-disciplines independently: that question
exceeds the scope of metrics and metric validation.

> Thus, I am bit doubtful about your claims of an "empirically
demonstrated
> fact."

Within the scope mentioned -- the RAE peer rankings, for disciplines
such as they have been partitioned for the past two decades -- there is
ample grounds for confidence in the empirical results to date.

(And please note that this has nothing to do with journal impact
factors,
journal field classification, or journal rankings. It is about the RAE
and the ranking of university departments by peer panels, as correlated
with citation counts.)

Stevan Harnad
AMERICAN SCIENTIST OPEN ACCESS FORUM:
http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-For
um.html
     http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/

UNIVERSITIES and RESEARCH FUNDERS:
If you have adopted or plan to adopt a policy of providing Open Access
to your own research article output, please describe your policy at:
     http://www.eprints.org/signup/sign.php
     http://openaccess.eprints.org/index.php?/archives/71-guid.html
     http://openaccess.eprints.org/index.php?/archives/136-guid.html

OPEN-ACCESS-PROVISION POLICY:
     BOAI-1 ("Green"): Publish your article in a suitable toll-access
journal
     http://romeo.eprints.org/
OR
     BOAI-2 ("Gold"): Publish your article in an open-access journal
if/when
     a suitable one exists.
     http://www.doaj.org/
AND
     in BOTH cases self-archive a supplementary version of your article
     in your own institutional repository.
     http://www.eprints.org/self-faq/
     http://archives.eprints.org/
     http://openaccess.eprints.org/