The Death of Peer Review? Rumour Premature...

[Stevan Harnad]

On Thu, 14 Dec 2006, Christopher D. Green wrote:

> Any comment?
> http://education.guardian.co.uk/RAE/story/0,,1969758,00.html
>
>   The death of peer review
>   Research notes
>   Natasha Gilbert
>   The Guardian 
>   Tuesday December 12, 2006
>
>   "The chancellor has decided to do away with the age-old, and trusted,
>   system of peer review for assessing the quality of science coming
>   out of the UK's universities - which has been used as the basis for
>   carving up public funding."

(1) Peer review of research publications is conducted by the referees consulted by
peer-reviewed journals.

(2) Peer review of competitive research grant applications is conducted
by the referees consulted by research funding councils.

(3) The RAE is neither a research journal nor a competitive research
grant funding council.

(4) The RAE is part of a *dual* research funding system: (i) competitive
research grant applications plus (ii) top-sliced funding based on RAE
ranking of each university department's research performance.

(5) The RAE panel review is not peer review, and never has been peer review: It
is a time-consuming, wasteful re-review of *already* peer-reviewed publications.

(6) "Metrics" are statistical indicators of research performance such as
publication counts, citations, downloads, links, students, funding, etc.

(7) Metrics are already highly correlated with RAE rankings.

(8) What has (at long last) been replaced by metrics is the
time-consuming, wasteful RAE panel re-review of *already* peer-reviewed
publications.

We should be celebrating the long overdue death of RAE panel re-review, not
prematurely feting the demise of peer review itself, which is alive and well.

A more worrisome question concerns *which* metrics will be used:

    "From 2010-11, science, engineering, technology and medicine (SET)
    subjects will instead be assessed using statistical indicators,
    such as the number of postgraduate students in a department and the
    amount of money a department brings in through its research."

The fallacy here is that the RAE is supposed to be part of a *dual* funding
system. If competitive funding is used as a heavily weighted metric, it is
tantamount to collapsing it all into just one system -- competitive grant
applications -- and merely increasing the amount of money given to the
winners: A self-fulfilling prophecy and a whopping "Matthew Effect."

Yet in the OA world there are a rich variety of potential metrics, which
should be tested and validated and customised to each discipline.

    Let 1000 RAE Metric Flowers Bloom: 
    Avoid Matthew Effect as Self-Fulfilling Prophecy
    http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/5418.html

    "Metrics" are Plural, Not Singular: Valid Objections From UUK About RAE
    http://openaccess.eprints.org/index.php?/archives/137-guid.html

>   "This new system should solve the much-complained-about bureaucracy of
>   the research assessment exercise (RAE). But some, such as the Royal
>   Society, the UK's academy of science, are adamant that sounding the
>   death-knell for peer review in SET subjects is a bad move."

Metrics will put an end to wasting UK researchers' time re-reviewing and being
re-reviewed, allowing them to devote their time instead to doing research. But a
biassed and blinkered choice of metrics will sound the death-knell of the dual
funding system (not peer review).

Stevan Harnad