Future UK RAEs to be Metrics-Based

[Stevan Harnad]

On Sat, 30 Sep 2006, David Goodman wrote:

> If X publishes a project chopped up into 4 short papers at about the 
> same time, if I'm working in the field I'll cite them all. 

(0) I can only repeat: if we want to give higher weight to average citations
than total citations, light-weight papers can be down-weighted.

> It is usual in the US for a tenure committee... to 
> limit the number of papers submitted...
> so they don't have to read them all...
> they really want to see the best work

(1) This is about UK national RAE, for departments, not US tenure review,
for individuals.

(2) This is about replacing RAE panel re-review by metrics. There is no
earthly reason *metrics* should be restricted to four papers (or
restricted in any way: the way to constrain metrics is via weightings).

> the fewer papers, the lower the publication 
> costs, no matter who... pays... I like...
> OAJournals [because] they for the first time provide a real 
> incentive to publish in larger chunks. From a large grant, 5 papers a year 
> at $3000 isn't much, but 20 would pay for another post-doc.  

(3) This is about OA self-archiving, not OA journals, and about RAE
submission via IRs.

Stevan Harnad

> ----- Original Message -----
> From: Stevan Harnad <harnad at ECS.SOTON.AC.UK>
> Date: Friday, September 29, 2006 5:33 am
> Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> To: SIGMETRICS at LISTSERV.UTK.EDU
> 
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> > 
> > On Fri, 29 Sep 2006, Sally Morris (Chief Executive) wrote:
> > 
> > > I disagree with Stevan about the advisability of limiting the 
> > number of
> > > papers which may be submitted.  Similar rules were introduced a 
> > few years
> > > ago in the US - the objective is to discourage authors from a 
> > 'never mind
> > > the quality, feel the width' mentality which can (and arguably 
> > does) lead to
> > > excessive publication, via salami-slicing and other (sometimes 
> > even less
> > > desirable - see http://www.alpsp.org/events/2006/PET/default.htm) 
> > methods
> > Sally (and no doubt many others) vastly under-rate the power of OA
> > metrics here: 
> > 
> > (1) Yes salami-slicing is bad.
> > 
> > (2) But metrics makes it easily detectable, and penalizable, by
> > differential weighting.
> > 
> > (3) Example: Researcher A receives a total of 100 citations for 10
> > papers, averaging 10 per paper; Researcher B receives a total of 100
> > citations for 4 papers, averaging 25 per paper. Easy to give the lower
> > average a lower weight; as a sub-test, easy to check the citations for
> > the top four papers too...
> > 
> > (4) The main idea is to stop wasting time and money re-submitting the
> > papers (to RAE) and re-reviewing them (by RAE panels) and let the
> > metrics do the work instead.
> > 
> > (5) Nor are citations count and averages and top-slicing near being
> > the only metrics that can enter into the weighted equation: There are
> > downloads, co-citations (what kind of research/researcher is it cited
> > *with*), authority metrics (what kind of research/researcher is it 
> > cited*by*), endogamy/exogamy metrics (how incestuous are the 
> > citations, in
> > the range: self-citations, co-author citations, mutual citation 
> > circles,within-specialty citations, interdisciplinarity), growth 
> > rate of citations
> > (and downloads), latency and longevity scores, etc.
> > 
> > (6) All of those metrics can be gathered and weighted, with the 
> > weights adjusted to the features of the field (some fields are 
> > rapid, narrow
> > growth, some are slower, broader growth, some more endogamous, some 
> > more exogamous, etc.).
> > 
> > (7) "Semantic" (in reality syntactic) content-based metrics can also
> > measure the degree of textual overlap between papers, both multiple
> > papers by the same author, and overlap with papers by other authors...
> > 
> > (8) And that just scratches the surface of OA metric possibilities.
> > 
> > My own understanding is that restricting RAE submissions to only 4
> > papers had been done partly to keep the work of the panel tractable
> > (mooted now with metrics) and partly to discourage salami-slicing --
> > but I know of no evidence whether it *did* discourage salami-slicing
> > (in either the UK or the US: does anyone have data?): After all, RAE
> > is not the only fish in the sea, for the author. (Objective 
> > evidence on
> > whether it had any effect, by the way, would have to be metric!)
> > 
> > But in a metric RAE, salami-slicing would become its own enemy, 
> > just as
> > self-citing and plagiarism would be. (And before you mention self-
> > padded downloads, that's readily detectable and name-shameable too, 
> > not only
> > by checking IPs but via triangulation with other metrics that are 
> > normally correlated with downloads, such as citations!)
> > 
> > Stevan Harnad
> > 
> > PS Whatever works in the UK, the US will eventually catch up too:
> > Metrics will be the measure in both cases, validated, as needed, 
> > against peer evaluation, the specific needs of a field, and internal 
> > validation through triangulation.
> > 
> > > ----- Original Message ----- 
> > > From: "Stevan Harnad" <harnad at ECS.SOTON.AC.UK>
> > > >
> > > > At the heart of this are not one, not two, not three, but 
> > *four* pieces
> > > > of patent nonsense so absurd as to take one's breath away. Most 
> > of the
> > > > nonsense is on RAE/HEFCE's end; one cannot blame the publishers 
> > for play
> > > > along (especially as the gentleman's agreement holds some hope of
> > > > forestalling OA a bit longer, or at least the role the RAE 
> > might have
> > > > played in hastening OA's arrival):
> > > >
> > > >    (1) The first piece of nonsense is the RAE's pedantic and
> > > >    dysfunctional insistence on laying their hands directly on the
> > > >    "originals," the publisher's version of each article per 
> > author,> >    rather than sensibly settling for the author's peer-
> > reviewed final
> > > >    drafts (postprints).
> > > >
> > > >    (2) The second is the equally foolish notion that the RAE 
> > somehow> >    needs special permission to do this, or, worse, might 
> > even have needed
> > > >    to *pay* for the right, but for this "gentleman's 
> > agreement"! (Of
> > > >    course the publishers are more than happy to play along with 
> > this> >    self-imposed farce on RAE's part; but if no one had ever 
> > absurdly> >    suggested in the first place that when an author 
> > sends a copy of his
> > > >    own paper to his own funder for evaluation, *he needs his 
> > publisher's> >    permission*, none of this nonsense would ever 
> > even have come up!)
> > > >
> > > >    (3) The idea of restricting submissions to only *four* papers
> > > >    was originally floated by RAE in part out of the hope that
> > > >    this limitation would act as a counterweight against salami-
> > sliced> >    publication. It didn't. And it's time to drop this 
> > absurd, arbitrary
> > > >    limit on what work can be submitted.
> > > >
> > > >    (4) Of course the other reason the number was kept down to 
> > four was
> > > >    the even more dysfunctional feature of the RAE that is only 
> > now,> >    at long last, being deservedly jettisoned (the 
> > submissions and panel
> > > >    reviews themselves!); yet one hand does not seem to be aware 
> > of what
> > > >    the other is doing: For once the unnecessary and time/money-
> > wasting> >    "peer-*re*-reviewing" that the RAE panels had been 
> > trying to
> > > >    do is at last abandoned in favour of metrics, there will be no
> > > >    need for either a 4-item cap or any compulsive attempt to 
> > get the
> > > >    "originals" to the panel. The authors' self-archived postprints
> > > >    in their own institutional OA IRs will suffice (and the only 
> > thing> >    the RAE panels -- if there still *are* any RAE panels --
> > need do,
> > > >    if suspicious about any particular item, is a database 
> > search (say,
> > > >    in Web of Knowledge or Scopus or PubMed) to make sure that 
> > the item
> > > >    in question did indeed appear in the journal indicated, 
> > under the
> > > >    name of the author in question).
> > > >
> > > > What will moot all of this is of course the OA self-archiving 
> > mandates> > by RCUK and the UK universities themselves, which will 
> > fill the UK
> > > > universities' IRs, which will in their turn -- with the help of 
> > the IRRA
> > > > http://irra.eprints.org/ -- mediate the submission of both the 
> > postprints> > and the metrics to the RAE. Then this ludicrous side-
> > show about the
> > > > "licensing" of the all-important "originals" to the RAE, for "peer
> > > > re-review" via the mediation of CrossRef and the publishers 
> > will at last
> > > > be laid to rest, once and for all.
> > > >
> > > > RAE 2008 will be its last hurrah...
> > > >
> > > > Stevan Harnad
> > > >
> > > > 
> > ______________________________________________________________________> > This email has been scanned by the 
> MessageLabs Email Security System.
> > > > For more information please visit http://www.messagelabs.com/email
> > > > 
> > ______________________________________________________________________> 
> > > 
> > 
>