Future UK RAEs to be Metrics-Based

[Stevan Harnad]

On Fri, 29 Sep 2006, Sally Morris (Chief Executive) wrote:

> I disagree with Stevan about the advisability of limiting the number of
> papers which may be submitted.  Similar rules were introduced a few years
> ago in the US - the objective is to discourage authors from a 'never mind
> the quality, feel the width' mentality which can (and arguably does) lead to
> excessive publication, via salami-slicing and other (sometimes even less
> desirable - see http://www.alpsp.org/events/2006/PET/default.htm) methods

Sally (and no doubt many others) vastly under-rate the power of OA
metrics here: 

(1) Yes salami-slicing is bad.

(2) But metrics makes it easily detectable, and penalizable, by
differential weighting.

(3) Example: Researcher A receives a total of 100 citations for 10
papers, averaging 10 per paper; Researcher B receives a total of 100
citations for 4 papers, averaging 25 per paper. Easy to give the lower
average a lower weight; as a sub-test, easy to check the citations for
the top four papers too...

(4) The main idea is to stop wasting time and money re-submitting the
papers (to RAE) and re-reviewing them (by RAE panels) and let the
metrics do the work instead.

(5) Nor are citations count and averages and top-slicing near being
the only metrics that can enter into the weighted equation: There are
downloads, co-citations (what kind of research/researcher is it cited
*with*), authority metrics (what kind of research/researcher is it cited
*by*), endogamy/exogamy metrics (how incestuous are the citations, in
the range: self-citations, co-author citations, mutual citation circles,
within-specialty citations, interdisciplinarity), growth rate of citations
(and downloads), latency and longevity scores, etc.

(6) All of those metrics can be gathered and weighted, with the weights
adjusted to the features of the field (some fields are rapid, narrow
growth, some are slower, broader growth, some more endogamous, some more
exogamous, etc.).

(7) "Semantic" (in reality syntactic) content-based metrics can also
measure the degree of textual overlap between papers, both multiple
papers by the same author, and overlap with papers by other authors...

(8) And that just scratches the surface of OA metric possibilities.

My own understanding is that restricting RAE submissions to only 4
papers had been done partly to keep the work of the panel tractable
(mooted now with metrics) and partly to discourage salami-slicing --
but I know of no evidence whether it *did* discourage salami-slicing
(in either the UK or the US: does anyone have data?): After all, RAE
is not the only fish in the sea, for the author. (Objective evidence on
whether it had any effect, by the way, would have to be metric!)

But in a metric RAE, salami-slicing would become its own enemy, just as
self-citing and plagiarism would be. (And before you mention self-padded
downloads, that's readily detectable and name-shameable too, not only
by checking IPs but via triangulation with other metrics that are normally
correlated with downloads, such as citations!)

Stevan Harnad

PS Whatever works in the UK, the US will eventually catch up too:
Metrics will be the measure in both cases, validated, as needed, against
peer evaluation, the specific needs of a field, and internal validation
through triangulation.

> ----- Original Message ----- 
> From: "Stevan Harnad" <harnad at ECS.SOTON.AC.UK>
> >
> > At the heart of this are not one, not two, not three, but *four* pieces
> > of patent nonsense so absurd as to take one's breath away. Most of the
> > nonsense is on RAE/HEFCE's end; one cannot blame the publishers for play
> > along (especially as the gentleman's agreement holds some hope of
> > forestalling OA a bit longer, or at least the role the RAE might have
> > played in hastening OA's arrival):
> >
> >    (1) The first piece of nonsense is the RAE's pedantic and
> >    dysfunctional insistence on laying their hands directly on the
> >    "originals," the publisher's version of each article per author,
> >    rather than sensibly settling for the author's peer-reviewed final
> >    drafts (postprints).
> >
> >    (2) The second is the equally foolish notion that the RAE somehow
> >    needs special permission to do this, or, worse, might even have needed
> >    to *pay* for the right, but for this "gentleman's agreement"! (Of
> >    course the publishers are more than happy to play along with this
> >    self-imposed farce on RAE's part; but if no one had ever absurdly
> >    suggested in the first place that when an author sends a copy of his
> >    own paper to his own funder for evaluation, *he needs his publisher's
> >    permission*, none of this nonsense would ever even have come up!)
> >
> >    (3) The idea of restricting submissions to only *four* papers
> >    was originally floated by RAE in part out of the hope that
> >    this limitation would act as a counterweight against salami-sliced
> >    publication. It didn't. And it's time to drop this absurd, arbitrary
> >    limit on what work can be submitted.
> >
> >    (4) Of course the other reason the number was kept down to four was
> >    the even more dysfunctional feature of the RAE that is only now,
> >    at long last, being deservedly jettisoned (the submissions and panel
> >    reviews themselves!); yet one hand does not seem to be aware of what
> >    the other is doing: For once the unnecessary and time/money-wasting
> >    "peer-*re*-reviewing" that the RAE panels had been trying to
> >    do is at last abandoned in favour of metrics, there will be no
> >    need for either a 4-item cap or any compulsive attempt to get the
> >    "originals" to the panel. The authors' self-archived postprints
> >    in their own institutional OA IRs will suffice (and the only thing
> >    the RAE panels -- if there still *are* any RAE panels -- need do,
> >    if suspicious about any particular item, is a database search (say,
> >    in Web of Knowledge or Scopus or PubMed) to make sure that the item
> >    in question did indeed appear in the journal indicated, under the
> >    name of the author in question).
> >
> > What will moot all of this is of course the OA self-archiving mandates
> > by RCUK and the UK universities themselves, which will fill the UK
> > universities' IRs, which will in their turn -- with the help of the IRRA
> > http://irra.eprints.org/ -- mediate the submission of both the postprints
> > and the metrics to the RAE. Then this ludicrous side-show about the
> > "licensing" of the all-important "originals" to the RAE, for "peer
> > re-review" via the mediation of CrossRef and the publishers will at last
> > be laid to rest, once and for all.
> >
> > RAE 2008 will be its last hurrah...
> >
> > Stevan Harnad
> >
> > ______________________________________________________________________
> > This email has been scanned by the MessageLabs Email Security System.
> > For more information please visit http://www.messagelabs.com/email
> > ______________________________________________________________________
> 
>