Future UK RAEs to be Metrics-Based

[David Goodman]

Problem is , Stevan, that though you ought to be right it does not always 
work out that way.  If X publishes a project chopped up into 
4 short papers at about the same time, if I'm working in the field I'll cite them all. 

It is usual in the US for a tenure committee in a first rate place to limit the 
number of papers submitted, and there are two reasons:
first, as you said, so they don't have to read them all--time is finite.
second, tho, is that they really want to see the best work, and judge from that,
not all the other papers that might have been published to give each grad 
student something for his CV.  

And it can be worse than salami slicing--it can be what I will call for want of 
a better metaphor salami squashing, which I define as saying the same thing 
is slightly differet ways for as many journals as you can--there is a limit, though, 
since eventually you run into the same referees. 

And there are side benefits: the fewer papers, the lower the publication costs,
no matter who it is that pays. One of the reasons I like a system of 
 OAJournals so much is that they for the first time provide a real incentive to publish in larger 
chunks. From a large grant, 5 papers a year at $3000 isn't much, but 20 
would pay for another post-doc.  

David Goodman, Ph.D., M.L.S.
previously:
Bibliographer and Research Librarian
Princeton University Library

dgoodman at princeton.edu 

----- Original Message -----
From: Stevan Harnad <harnad at ECS.SOTON.AC.UK>
Date: Friday, September 29, 2006 5:33 am
Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
To: SIGMETRICS at LISTSERV.UTK.EDU

> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
> 
> On Fri, 29 Sep 2006, Sally Morris (Chief Executive) wrote:
> 
> > I disagree with Stevan about the advisability of limiting the 
> number of
> > papers which may be submitted.  Similar rules were introduced a 
> few years
> > ago in the US - the objective is to discourage authors from a 
> 'never mind
> > the quality, feel the width' mentality which can (and arguably 
> does) lead to
> > excessive publication, via salami-slicing and other (sometimes 
> even less
> > desirable - see http://www.alpsp.org/events/2006/PET/default.htm) 
> methods
> Sally (and no doubt many others) vastly under-rate the power of OA
> metrics here: 
> 
> (1) Yes salami-slicing is bad.
> 
> (2) But metrics makes it easily detectable, and penalizable, by
> differential weighting.
> 
> (3) Example: Researcher A receives a total of 100 citations for 10
> papers, averaging 10 per paper; Researcher B receives a total of 100
> citations for 4 papers, averaging 25 per paper. Easy to give the lower
> average a lower weight; as a sub-test, easy to check the citations for
> the top four papers too...
> 
> (4) The main idea is to stop wasting time and money re-submitting the
> papers (to RAE) and re-reviewing them (by RAE panels) and let the
> metrics do the work instead.
> 
> (5) Nor are citations count and averages and top-slicing near being
> the only metrics that can enter into the weighted equation: There are
> downloads, co-citations (what kind of research/researcher is it cited
> *with*), authority metrics (what kind of research/researcher is it 
> cited*by*), endogamy/exogamy metrics (how incestuous are the 
> citations, in
> the range: self-citations, co-author citations, mutual citation 
> circles,within-specialty citations, interdisciplinarity), growth 
> rate of citations
> (and downloads), latency and longevity scores, etc.
> 
> (6) All of those metrics can be gathered and weighted, with the 
> weightsadjusted to the features of the field (some fields are 
> rapid, narrow
> growth, some are slower, broader growth, some more endogamous, some 
> moreexogamous, etc.).
> 
> (7) "Semantic" (in reality syntactic) content-based metrics can also
> measure the degree of textual overlap between papers, both multiple
> papers by the same author, and overlap with papers by other authors...
> 
> (8) And that just scratches the surface of OA metric possibilities.
> 
> My own understanding is that restricting RAE submissions to only 4
> papers had been done partly to keep the work of the panel tractable
> (mooted now with metrics) and partly to discourage salami-slicing --
> but I know of no evidence whether it *did* discourage salami-slicing
> (in either the UK or the US: does anyone have data?): After all, RAE
> is not the only fish in the sea, for the author. (Objective 
> evidence on
> whether it had any effect, by the way, would have to be metric!)
> 
> But in a metric RAE, salami-slicing would become its own enemy, 
> just as
> self-citing and plagiarism would be. (And before you mention self-
> paddeddownloads, that's readily detectable and name-shameable too, 
> not only
> by checking IPs but via triangulation with other metrics that are 
> normallycorrelated with downloads, such as citations!)
> 
> Stevan Harnad
> 
> PS Whatever works in the UK, the US will eventually catch up too:
> Metrics will be the measure in both cases, validated, as needed, 
> againstpeer evaluation, the specific needs of a field, and internal 
> validationthrough triangulation.
> 
> > ----- Original Message ----- 
> > From: "Stevan Harnad" <harnad at ECS.SOTON.AC.UK>
> > >
> > > At the heart of this are not one, not two, not three, but 
> *four* pieces
> > > of patent nonsense so absurd as to take one's breath away. Most 
> of the
> > > nonsense is on RAE/HEFCE's end; one cannot blame the publishers 
> for play
> > > along (especially as the gentleman's agreement holds some hope of
> > > forestalling OA a bit longer, or at least the role the RAE 
> might have
> > > played in hastening OA's arrival):
> > >
> > >    (1) The first piece of nonsense is the RAE's pedantic and
> > >    dysfunctional insistence on laying their hands directly on the
> > >    "originals," the publisher's version of each article per 
> author,> >    rather than sensibly settling for the author's peer-
> reviewed final
> > >    drafts (postprints).
> > >
> > >    (2) The second is the equally foolish notion that the RAE 
> somehow> >    needs special permission to do this, or, worse, might 
> even have needed
> > >    to *pay* for the right, but for this "gentleman's 
> agreement"! (Of
> > >    course the publishers are more than happy to play along with 
> this> >    self-imposed farce on RAE's part; but if no one had ever 
> absurdly> >    suggested in the first place that when an author 
> sends a copy of his
> > >    own paper to his own funder for evaluation, *he needs his 
> publisher's> >    permission*, none of this nonsense would ever 
> even have come up!)
> > >
> > >    (3) The idea of restricting submissions to only *four* papers
> > >    was originally floated by RAE in part out of the hope that
> > >    this limitation would act as a counterweight against salami-
> sliced> >    publication. It didn't. And it's time to drop this 
> absurd, arbitrary
> > >    limit on what work can be submitted.
> > >
> > >    (4) Of course the other reason the number was kept down to 
> four was
> > >    the even more dysfunctional feature of the RAE that is only 
> now,> >    at long last, being deservedly jettisoned (the 
> submissions and panel
> > >    reviews themselves!); yet one hand does not seem to be aware 
> of what
> > >    the other is doing: For once the unnecessary and time/money-
> wasting> >    "peer-*re*-reviewing" that the RAE panels had been 
> trying to
> > >    do is at last abandoned in favour of metrics, there will be no
> > >    need for either a 4-item cap or any compulsive attempt to 
> get the
> > >    "originals" to the panel. The authors' self-archived postprints
> > >    in their own institutional OA IRs will suffice (and the only 
> thing> >    the RAE panels -- if there still *are* any RAE panels --
> need do,
> > >    if suspicious about any particular item, is a database 
> search (say,
> > >    in Web of Knowledge or Scopus or PubMed) to make sure that 
> the item
> > >    in question did indeed appear in the journal indicated, 
> under the
> > >    name of the author in question).
> > >
> > > What will moot all of this is of course the OA self-archiving 
> mandates> > by RCUK and the UK universities themselves, which will 
> fill the UK
> > > universities' IRs, which will in their turn -- with the help of 
> the IRRA
> > > http://irra.eprints.org/ -- mediate the submission of both the 
> postprints> > and the metrics to the RAE. Then this ludicrous side-
> show about the
> > > "licensing" of the all-important "originals" to the RAE, for "peer
> > > re-review" via the mediation of CrossRef and the publishers 
> will at last
> > > be laid to rest, once and for all.
> > >
> > > RAE 2008 will be its last hurrah...
> > >
> > > Stevan Harnad
> > >
> > > 
> ______________________________________________________________________> > This email has been scanned by the 
MessageLabs Email Security System.
> > > For more information please visit http://www.messagelabs.com/email
> > > 
> ______________________________________________________________________> 
> > 
>