Let 1000 RAE Metric Flowers Bloom: Avoid Matthew Effect as Self-Fulfilling Prophecy

[Stephen J Bensman]

That's not a good saying to launch your RAE metric reform.  Below is the
origins of your campaign slogan.  You will note that it is a quote from
Chairman Mao.    If it follows its model, it can lead to the forced
rustication of numerous British academics.

SB


 1956-7 Hundred Flowers and Anti-rightist Campaigns. As part of the effort to encourage the participation of
 intellectuals in the new regime, cultural and intellectual figures were encouraged to speak their minds on the state
 of CCP rule and programs. Mao personally took the lead in the movement, which was launched under the classical
 slogan "Let a hundred flowers bloom, let the hundred schools of thought contend". At first the party's repeated
 invitation to air constructive views freely and openly was met with caution. By mid-1957, however, the movement
 unexpectedly mounted, bringing denunciation and criticism against the party in general and the excesses of its
 cadres in particular. Startled and embarrassed, leaders turned on the critics as "bourgeois rightists" and launched
 the Anti-Rightist Campaign, resulting in a year-long purge of party members, sending officials "down" to the
 countryside (xiafang), and persecuting intellectuals who spoke out against the party.








Stevan Harnad <harnad at ECS.SOTON.AC.UK>@LISTSERV.UTK.EDU> on 06/21/2006
06:10:32 AM

Please respond to ASIS&T Special Interest Group on Metrics
       <SIGMETRICS at LISTSERV.UTK.EDU>

Sent by:    ASIS&T Special Interest Group on Metrics
       <SIGMETRICS at LISTSERV.UTK.EDU>


To:    SIGMETRICS at LISTSERV.UTK.EDU
cc:     (bcc: Stephen J Bensman/notsjb/LSU)

Subject:    [SIGMETRICS] Let 1000 RAE Metric Flowers Bloom: Avoid Matthew
       Effect as Self-Fulfilling Prophecy


Comment on:

    UK plans research funding overhaul
    by Stephen Pinfield
    The Scientist, Wednesday 21 June, 2006
    http://www.the-scientist.com/news/display/23683/

-----------------------------------------------------------------------
    Let 1000 RAE Metric Flowers Bloom:
    Avoid Matthew Effect as Self-Fulfilling Prophecy

    by Stevan Harnad

The conversion of the UK Research Assessment Exercise (RAE) from the
present costly, wasteful exercise to time-saving and cost-efficient
metrics is welcome and overdue, but the worrying thing is that
the RAE planners currently seem to be focused on just one metric --
prior research funding -- instead of the full and rich spectrum of new
(and old) metrics that will become available in an Open Access world,
with all the research performance data digitally available online
for analysis and use.

Mechanically basing the future RAE rankings exclusively on prior funding
would just generate a Matthew Effect (making the rich richer and the
poor poorer), a self-fulfilling prophecy that is simply equivalent to
increasing the amount given to those who were previously funded (and
scrapping the RAE altogether, as a separate, semi-independent performance
evaluator and funding source).

What the RAE *should* be planning to do is to look at weighted
combinations of all available research performance metrics -- including
the many that are correlated, but not so tightly correlated, with prior
RAE rankings, such as author/article/book citation counts, article
download counts, co-citations (co-cited with and co-cited by, weighted
with the citation weight of the co-citer/co-citee), endogamy/exogamy
metrics (citations by self or collaborators versus others, within
and across disciplines), hub/authority counts (in-cites and out-cites,
weighted recursively by the citation's own in-cite and out-cite counts),
download and citation growth rates, semantic-web correlates, etc.

It would be both arbitrary and absurd to blunt the sensitivity, power,
predictivity and validity of metrics a priori by biasing them toward the
prior-funding counts metric alone, which should just be one out of a full
battery of weighted metrics, adjusted to each discipline and validated
against one another (and against human judgment too).

    Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006) The Open
    Research Web: A Preview of the Optimal and the Inevitable, in Jacobs,
    N., Eds. Open Access: Key Strategic, Technical and Economic Aspects,
    chapter 21. Chandos.
    http://eprints.ecs.soton.ac.uk/12369/

------------------------------------------------------------------------
Stevan Harnad
AMERICAN SCIENTIST OPEN ACCESS FORUM:
A complete Hypermail archive of the ongoing discussion of providing
open access to the peer-reviewed research literature online (1998-2005)
is available at:
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/
        To join or leave the Forum or change your subscription address:
http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html

        Post discussion to:
        american-scientist-open-access-forum at amsci.org

UNIVERSITIES: If you have adopted or plan to adopt an institutional
policy of providing Open Access to your own research article output,
please describe your policy at:
        http://www.eprints.org/signup/sign.php

UNIFIED DUAL OPEN-ACCESS-PROVISION POLICY:
    BOAI-1 ("green"): Publish your article in a suitable toll-access
journal
            http://romeo.eprints.org/
OR
    BOAI-2 ("gold"): Publish your article in a open-access journal if/when
            a suitable one exists.
            http://www.doaj.org/
AND
    in BOTH cases self-archive a supplementary version of your article
            in your institutional repository.
            http://www.eprints.org/self-faq/
            http://archives.eprints.org/
            http://openaccess.eprints.org/