Future UK RAEs to be Metrics-Based

[Stevan Harnad]

On Wed, 20 Sep 2006, Andrew A. Adams wrote:

> What will be the impact amongst UK academics of making their funding (partly)
> dependent on their citation rates?

I think you may be misunderstanding the finding: UK academics' funding is *already*
"dependent" on citation rates, inasmuch as RAE rankings are *already* highly
correlated with citation rates -- and this, despite the fact that citation rates are
not explicitly counted by RAE panels currently, because RAE specifically forbids
it!

> It is the "law of unintended consequence" or, alternately, the lack of
> consideration about "game playing" that makes the current RAE such a bad
> method of deciding funding allocation. 

Without prejudice as to the reliability or validity of the current RAE's
rankings, such as they are, what makes the current RAE's grotesquely
onerous submission and panel-review process such a bad method is that
it wastes so much of UK researchers' time in submitting and reviewing
research that has already been submitted to and reviewed by peer-reviewed
journals -- instead of leaving UK researchers that time to do research!

Metrics -- already highly correlated with the RAE rankings in all fields tested to
date -- will remedy this.

> if you tell academics the rules of a
> game, then they will generally be quite good at playing that game to win.
> This skews the activity of academics away from their proper business unless
> you can show that the way to win the game is to do the "correct" things
> anyway.

The time/money-wasting RAE process has already skewed academics away from their
proper business (research and teaching) toward preparing their RAE returns. 

If what you mean is that metrics will breed abuses, you are right, but it will also
breed powerful ways of detecting and deterring those abuses. Open Access metrics are
accessible and analysable, openly, and computationally, by all. If someone tampers
with the data (excessive self-citation or collaborator circle-citation, or excessive
text overlap, or salami-slicing, or robotic download-padding, or tampering with
text), that is all detectable, name-and-shameable, and penalizable -- which will all
act as a deterrent, especially after the first few culprits are exposed.

In contrast, the RAE's needless and profligate paranoia about getting
(and licensing!) the exact photocopies of journal articles -- lest authors
submit false or doctored texts -- was absolutely absurd, and again, the
remedy would have been to consult official databases for confirmation
if in doubt about authorship or authenticity, and relying on metrics
instead of re-review. (And the ones with the biggest interest in policing
against malfeasance are the authors and their own institutions, for they
are the ones who stand to lose the most from being named-and-shamed,
not the RAE or HEFCE.)

> Now, citation counts are derived from worldwide academia, but remember that
> various other countries are also considering RAE-style measurements and some
> are looking at metrics, while others seem to be mostly following the UK
> example so we can assume they may well move to a similar metric-based system
> sometime in the future.

Agreed. (But it is not clear what your point here is: Heaven forfend that the
rest of the world should ape, instead, the bad old panel-RAE!)

> It's all very well to claim that without citation-metric-based funding
> decisions, that citation metrics mirror existing measurements. But what
> evidence is there that this is a robust relationship under the assumption
> that we move to a citation-metric measurement and away from the current system?

I couldn't follow quite what this meant, except if it was that currently,
with panel review merely incidentally correlated with metrics, there
is no direct incentive for metric abuse, whereas with direct reliance
on metrics, metric abuse is likely to be enhanced. You are quite right
about that, but the reply is as above: Metric abuse will also be far
more detectable, and punishable, which should provide the necessary
counter-incentive.

Ceterum censeo: Metrics does not just mean citation counts -- and it definitely does
not mean just prior-funding counts!

> Note: this discussion may well be veering too far off-topic from OA policy
> issues.

Slightly. But Open Access to text is inexorably linked to Open Access to the text
metrics.

Stevan Harnad
http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html