Citation statistics

Mon Jun 16 13:40:12 EDT 2008

Dear Stevan, 

If I correctly understand then your claim is that ranking results based on
peer review at the departmental level correlate highly with MEAN
departmental citation rates. This would be the case for psychology, wouldn't
it? 

It is an amazing result because one does not expect citation rates to be
normally distributed. (The means of the citation rates, of coures, are
normally distributed.) In my own department, for example (in communication
studies), we have various communities (social-psychologists, information
scientists, political science) with very different citation patterns. But
perhaps, British psychology departments are exceptionally homongeneous both
internally and comparatively. 

Then, you wish to strengthen this correlation by adding more indicators. The
other indicators may correlate better or worse with the ratings. The former
ones can add to the correlations, while the latter would worsen them. Or do
you wish only to add indicators which improve the correlations with the
ratings? 

I remember from a previous conversation on this subject that you have a kind
of multi-variate regression model in mind in which the RAE ratings would be
the dependent variable. One can make the model fit to the rankings by
estimating the parameters. One can also refine this per discipline. Would
one expecat any predictive power in such a model in a new situation (after 4
years)? Why?

With best wishes, 

Loet

________________________________

Loet Leydesdorff 
Amsterdam School of Communications Research (ASCoR), 
Kloveniersburgwal 48, 1012 CX Amsterdam. 
Tel.: +31-20- 525 6598; fax: +31-20- 525 3681 
loet at leydesdorff.net ; http://www.leydesdorff.net/ 

> -----Original Message-----
> From: ASIS&T Special Interest Group on Metrics 
> [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stevan Harnad
> Sent: Monday, June 16, 2008 3:20 PM
> To: SIGMETRICS at LISTSERV.UTK.EDU
> Subject: Re: [SIGMETRICS] Citation statistics
> 
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
> 
> On Sun, 15 Jun 2008, Loet Leydesdorff wrote:
> 
> >> SH: But what all this valuable, valid cautionary 
> discussion overlooks is not
> >> only the possibility but the *empirically demonstrated 
> fact* that there
> >> exist metrics that are highly correlated with human expert 
> rankings.
> >
> > It seems to me that it is difficult to generalize from one 
> setting in which
> > human experts and certain ranks coincided to the *existence *of such
> > correlations across the board. Much may depend on how the 
> experts are
> > selected. I did some research in which referee reports did 
> not correlate
> > with citation and publication measures.
> 
> Much may depend on how the experts are selected, but that was just as
> true during the 20 years in which rankings by experts were the sole
> criterion for the rankings in the UR Research Assessment Exercise
> (RAE). (In validating predictive metrics one must not endeavor to be
> Holier than the Pope: Your predictor can at best hope to be 
> as good as,
> but not better than, your criterion.)
> 
> That said: All correlations to date between total departmental author
> citation counts (not journal impact factors!) and RAE peer rankings
> have been positive, sizable, and statistically significant for the
> RAE, in all disciplines and all years tested. Variance there will be,
> always, but a good-sized component from citations alone seems to be
> well-established. Please see the studies of Professor Oppenheim and
> others, for example as cited in:
> 
>     Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) 
> Mandated online
>     RAE CVs Linked to University Eprint Archives: Improving 
> the UK Research
>     Assessment Exercise whilst making it cheaper and easier. 
> Ariadne 35.
>     http://www.ariadne.ac.uk/issue35/harnad/
> 
> > Human experts are necessarily selected from a population of 
> experts, and it
> > is often difficult to delineate between fields of expertise.
> 
> Correct. And the RAE rankings are done separately, discipline by
> discipline; the validation of the metrics should be done that way too.
> 
> Perhaps there is sometimes a case for separate rankings even at
> sub-disciplinary level. I expect the departments will be able to sort
> that out. (And note that the RAE correlations do not constitute a
> validation of metrics for evaluating individuals: I am confident that
> that too will be possible, but it will require many more metrics and
> much more validation.)
> 
> > Similarly, we
> > know from quite some research that citation and publication 
> practices are
> > field-specific and that fields are not so easy to 
> delineate. Results may be
> > very sensitive to choices made, for example, in terms of 
> citation windows.
> 
> As noted, some of the variance in peer judgments will depend on the
> sample of peers chosen; that is unavoidable. That is also why "light
> touch" peer re-validation, spot-checks, updates and 
> optimizations on the
> initialized metric weights are also a good idea, across the years.
> 
> As to the need to evaluate sub-disciplines independently: 
> that question
> exceeds the scope of metrics and metric validation.
> 
> > Thus, I am bit doubtful about your claims of an 
> "empirically demonstrated
> > fact."
> 
> Within the scope mentioned -- the RAE peer rankings, for disciplines
> such as they have been partitioned for the past two decades 
> -- there is
> ample grounds for confidence in the empirical results to date.
> 
> (And please note that this has nothing to do with journal 
> impact factors,
> journal field classification, or journal rankings. It is about the RAE
> and the ranking of university departments by peer panels, as 
> correlated
> with citation counts.)
> 
> Stevan Harnad
> AMERICAN SCIENTIST OPEN ACCESS FORUM:
> http://amsci-forum.amsci.org/archives/American-Scientist-Open-
> Access-Forum.html
>      http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/
> 
> UNIVERSITIES and RESEARCH FUNDERS:
> If you have adopted or plan to adopt a policy of providing Open Access
> to your own research article output, please describe your policy at:
>      http://www.eprints.org/signup/sign.php
>      http://openaccess.eprints.org/index.php?/archives/71-guid.html
>      http://openaccess.eprints.org/index.php?/archives/136-guid.html
> 
> OPEN-ACCESS-PROVISION POLICY:
>      BOAI-1 ("Green"): Publish your article in a suitable 
> toll-access journal
>      http://romeo.eprints.org/
> OR
>      BOAI-2 ("Gold"): Publish your article in an open-access 
> journal if/when
>      a suitable one exists.
>      http://www.doaj.org/
> AND
>      in BOTH cases self-archive a supplementary version of 
> your article
>      in your own institutional repository.
>      http://www.eprints.org/self-faq/
>      http://archives.eprints.org/
>      http://openaccess.eprints.org/
> 