Future UK RAEs to be Metrics-Based

[Stevan Harnad]

On Tue, 19 Sep 2006 l.hurtado at ED.AC.UK wrote:

> --Humanities scholarly publishing is more diverse in venue/genre than
> in some other fields.  Indeed, journals are not particularly regarded
> as quite so central, but only one among several respected and
> frequented genres, which include multi-author books, and (perhaps
> particularly) monographs.

Citation counts can in principle -- and up to a point already do --
count citations to all these genres: 

    (1) citations *from* articles *to* preprints, articles, chapters, and
    books (already being partially indexed, e.g., by ISI)

    (2) citations *from* preprints, articles, chapters, books *to*
    preprints, articles, chapters, and books (indexable in principle,
    already partly indexed by citebase, citeseer, google scholar and
    scopus, and will flourish dramatically once Open Access prevails)

Hence whatever statistically significant RAE/citation correlations
and effect sizes Charles Oppenheim manages to find *despite* the weak and
partial citation coverage to date is actually evidence of the robustness
of the RAE/citation correlation in the fields that are less article-based.

> QUESTION:  Are the studies that supposedly show such meaningful
> correlations actually drawing upon the full spread of publication
> genres appropriate to the fields in view?  

Not yet, and that's the point: Charles's findings are all the more
remarkable for being so robust, despite the weak signal!

> (I'd be surprised but
> delighted were the answer yes, because I'm not aware of any mechanism
> in place, such as ISI in journal monitoring, for surveying and counting
> in such a vast body of material.

The point is that citation coverage right now is most definitely
incomplete and insufficient. But that can (and will) only improve
(especially under pressure from the RAE, and OA!). Meanwhile, though,
the successful demonstration of strong correlations even based on the
partial coverage is very promising evidence.

> I'm not pushing at all for the labour-intensive RAE of the past.

Bravo. That means 80% agreement already!

> Indeed, if the question is not how do individual scholars stack up in
> comparison to others in their field (which the RAE actually wasn't
> designed to determine), but instead how can we identify depts into
> which a disproportionate amount of govt funding should be pumped, then
> I think in almost any field a group of informed scholars could readily
> determine the top 5-10 places within 30 minutes, and with time left
> over for coffee.

For all the UK departments, and in fair proportion to their merits and needs?
Or just for a familiar few?

And would several such informed-scholar circles agree on their rankings
(with one another, or with the current RAE rankings)?

> I'm just asking for more transparency and evidence behind the
> enthusiasm for replacing RAE with "metrics".

A group of informed scholars over coffee does not strike me as the height of
transparency and evidence...

However, in validating the new weighted metric equation, and adjusting
it for the needs of each discipline, one of the criteria against which
it will be validated is of course informed peer judgments: The metric
equation should not be at odds with informed peer judgment (nor should
there be marked discrepancies among metrics themselves, at least among
those we assume to be measuring the same sort of thing, such as downloads
and citations).

In general, with multiple regression equations (which, by the way,
capture only linear effects, unless orthogonal polynomials guessing at
nonlinear relations are used), one wants the measures that are meant to
measure the same sort of thing to be correlated with one another, but one
does not want the correlation to be *too* high, otherwise the measures
are redundant: Optimally, they should be cross-checks on one another, but
also each should be making its own unique contribution to the prediction,
over and above corroborating the rest. And of course the weight of each
should be adjustable in accordance with the specific profile of the
discipline and its needs and values.

For example, exogamy might be more of a virtue in some fields than
others. Some fields may be more authority-based or co-citation
authority-based than others. For some fields, steep early uptake may be
predictive, for others, longevity. etc. This will all be brought into
focus by the metric validation and calibration and customization
phase that will have to precede the use of the scientometric equation
for evaluation -- exactly as validation, standardization and the
creation of norms and benchmarks must be done in biometrics and
psychometrics before using the metrics for clinical or evaluative
purposes.

We are talking about a rich new OA world of online performance
indicators and predictors sitting on top of an even richer primary
database: the research itself.

    Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006) The Open
    Research Web: A Preview of the Optimal and the Inevitable, in Jacobs,
    N., Eds. Open Access: Key Strategic, Technical and Economic Aspects,
    chapter 20. Chandos.   http://eprints.ecs.soton.ac.uk/12453/

Stevan Harnad

> Larry
> 
> Quoting "C.Oppenheim" <C.Oppenheim at LBORO.AC.UK>:
> 
> > The correlation is between number of citations in total (and average number
> > of citations per member of staff) received by a Department over the RAE
> > period (1996-2001) and the RAE score received by the Department following
> > expert peer review.  Correlation analyses are done using Pearson or Spearman
> > correlation coefficients. The fact that so few humanities scholars publish
> > journal articles does not affect this result.
> >
> > A paper on the topic  is in preparation at the moment.
> >
> > What intrigues me is why there is so much scepticism about the notion.  RAE
> > is done by peer review experts.  Citations are also done by (presumably)
> > experts who choose to cite a particular work.  So one would expect a
> > correlation between the two, wouldn't one?  What it tells us is that high
> > quality research leads to both high RAE scores AND high citation counts.
> >
> >  I do these calculations (and I've covered many subject areas over the
> > years, but not biblical studies - something for the future!) in a totally
> > open-minded manner.  If I get a non-significant or zero correlation in such
> > a study in the future, I will faithfully report it.  But so far, that hasn't
> > happened.
> >
> > Charles
> >
> > Professor Charles Oppenheim
> > Head
> > Department of Information Science
> > Loughborough University
> > Loughborough
> > Leics LE11 3TU
> >
> > Tel 01509-223065
> > Fax 01509-223053
> > e mail C.Oppenheim at lboro.ac.uk
> > ----- Original Message -----
> > From: <l.hurtado at ED.AC.UK>
> > To: <AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM at LISTSERVER.SIGMAXI.ORG>
> > Sent: Monday, September 18, 2006 8:37 PM
> > Subject: Re: Future UK RAEs to be Metrics-Based
> >
> >
> School of Divinity, New College
> University of Edinburgh
> Mound Place
> Edinburgh, UK. EH1 2LX
> Office Phone:  (0)131 650 8920. FAX:  (0)131 650 7952
>