Future UK RAEs: Peer review or Metrics-Based

[Stevan Harnad]

On Sun, 9 Apr 2006, Jonathan Levitt wrote:

 > In principle, the sequel to the RAE could take into account the
 > accrual reviews from the journal that published the article
 > (rather than conduct their own reviews), but in my view there
 > would need to be stringent checks on their authenticity.

Submit the referee reports from the journals in which the articles
were published? No harm in that, I suppose, but what on earth for?
The journals did the peer review, and the attestation to that fact is
the published article, the journal name, and the journal's
established track record for quality. The rest is down to metrics
(journal impact factors, author/article citation counts, downloads,
co-citation fan-in/out quality, recursively weighted authority
CiteRank, latency/longevity, co-text, etc. etc.). After 25 years of
editing a very high impact peer-reviewed journal, Behavioral and
Brain Sciences (BBS) -- http://www.bbsonline.org/ -- I cannot see any
added benefit (though no harm) from forwarding the referee reports --
and, presumably, the editorial disposition letter -- to a tenure/
promotion committee or RAE panel. The peer-review has already
performed its function in getting the article suitably revised,
accepted, and tagged with the journal's established quality-standard.
The referee reports are informative to the editor, but not to others.
On the other hand, the next phase -- which my own journal, BBS,
pursued very actively and explicitly, namely open peer commentary --
would be extremely informative for those with the time, patience and
expertise to read and weigh it. Otherwise, there too, commentary
metrics, including commentary-content metrics (+/-/=) will without
the slightest doubt emerge from an Open Access full-text database.

http://www.ecs.soton.ac.uk/%7Eharnad/Temp/Kata/bbs.editorial.html
http://www.ecs.soton.ac.uk/%7Eharnad/Temp/bbs.valedict.html

Stevan Harnad


On 9-Apr-06, at 10:02 AM, Jonathan Levitt wrote:
>
> Steven wrote "1) peer review has already been done for published
> articles,
> so the issue is not (i) peer review vs. metrics but (ii)  peer
> review plus
> metrics vs. peer review plus metrics plus 'peer re-review' (by the RAE
> panels)."  .  To me the issue is not so much peer review vs.
> metrics, but
> finding a combination of peer review and citation/usage metrics
> that seems
> particularly likely to be effective at measuring research quality.
>
> In principle, the sequel to the RAE could take into account the
> accrual
> reviews from the journal that published the article (rather than
> conduct
> their own reviews), but in my view there would need to be stringent
> checks
> on their authenticity.
>
> Best regards,
> Jonathan.
>
>
> ----- Original Message -----
> From: "Stevan Harnad" <harnad at ECS.SOTON.AC.UK>
> To: <SIGMETRICS at LISTSERV.UTK.EDU>
> Sent: Thursday, April 06, 2006 10:56 PM
> Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
>
>
>> Adminstrative info for SIGMETRICS (for example unsubscribe):
>> http://web.utk.edu/~gwhitney/sigmetrics.html
>>
>> The following article is excellent and accurate overall.
>>
>>    Cliffe, Rebeca (2006) Research Assessment Exercise: Bowing
>>    out in Favour of Metrics. EPS Insights: 3 April 2006
>>    http://technology.guardian.co.uk/weekly/story/0,,1747334,00.html
>>
>> One can hardly quarrel with the following face-valid summary from
>> this article:
>>
>>>   "The move to a new metrics based system [for RAE] will no doubt
>>> please
>>>   those who see a role for institutional repositories in monitoring
>>>   research quality. The online environment has thrown up new
>>> metrics,
>>>   which could be used alongside traditional measures such as
>>> citations.
>>>   Usage can be measured at the point of consumption -- the number
>>>   of "hits" on a particular article can indicate the uptake of the
>>>   research. Web usage would be expected to be an early indicator of
>>>   how often the article is later cited. Some believe that
>>> institutional
>>>   repositories should be used as the basis for ongoing assessment
>>> of all
>>>   UK peer-reviewed research output by mandating that researchers
>>> should
>>>   place material in repositories. They argue that this would allow
>>>   usage to be measured earlier, through downloads of both pre-prints
>>>   and post-prints. Of course, this course of action would also
>>> advance
>>>   the cause of open access by making this research available free."
>>
>> But there are a few points of detail on which this otherwise accurate
>> report could be made even more useful:
>>
>>    (1) peer review has already been done for published articles, so
>>    the issue is not (i) peer review vs. metrics but (ii)  peer review
>>    plus metrics vs. peer review plus metrics plus "peer re-review"
>>    (by the RAE panels). It is the re-review of already peer-reviewed
>>    publications that is the wasteful practice that needs to be
>> scrapped,
>>    given that peer review has already been done, and that metrics are
>>    already highly correlated with the RAE ranking outcome anyway.
>>
>>    (2) For the fields in which the current RAE outcome is not already
>>    highly correlated with metrics, further work is needed; obviously
>>    works other than peer-reviewed article or books (e.g., artwork,
>>    multimedia) will have to be evaluated in other ways, but for
>>    science, engineering, biomedicine, social science, and most fields
>>    of humanities, books and articles are the form that research
>> output
>>    takes, and they will be amenable to the increasingly powerful and
>>    diverse forms of metrics that are being devised and tested. (Many
>>    will be tested in parallel with the 2008 RAE, which will still be
>>    conducted the old, wasteful way; some of the metrics may also be
>>    testable retrospectively against prior RAE outcomes.)
>>
>>>   "Proponents of a metrics based system point to studies that show
>>>   how average citation frequencies of articles can closely predict
>>>   the scores given by the RAE for departmental quality, even though
>>>   the RAE does not currently count these."
>>
>> True, but the highest metric correlate of the present RAE outcome
>> is reportedly prior research funding (0.98). Yet it would be a big
>> mistake to scrap all other metrics and base the RAE rank on just
>> prior funding. That would just generate a massive Matthew Effect and
>> essentially make top-sliced RAE funding redundant with direct
>> competitive
>> research project funding (thereby essentially "bowing out" of the
>> dual
>> RAE/RCUK funding system altogether, reducing it to just research
>> project
>> funding). What is remarkable about the high correlation between
>> citation
>> counts and RAE ranks (0.7 - 0.9), even though the correlation is
>> not quite
>> as high as with prior funding (0.98), is that citations are not
>> presently
>> counted in the RAE (whereas prior funding is)! Not only are
>> citations a
>> more independent metric of research performance than prior
>> funding, but
>> counting them directly -- along with the many other candidate
>> metrics --
>> can enrich and diversify the RAE evaluation, rather than just make it
>> into a self-fulfilling prophecy.
>>
>>>   "However, metrics tend to work better for the sciences than the
>>>   humanities. Whereas the citation of science research is seen as an
>>>   indicator of the quality and impact of the research, in the
>>> humanities
>>>   this is not the case.  Humanities research is based around
>>> critical
>>>   discourse and an author may be citing an article simply to
>>> disagree
>>>   with its argument."
>>
>> I don't think this is quite accurate. It might be true that
>> humanities
>> research makes less explicit use of citation counts today than
>> science
>> research. It might even be true that the correlation between citation
>> counts and research productivity and importance is lower in the
>> humanities than in the sciences (though I am not aware of studies
>> to that
>> effect). And it may also be true that citation counts in the
>> humanities
>> are less correlated with RAE rankings than they are in the
>> sciences. But
>> the familiar canard about articles being cited, not because they are
>> valid but important, but in order to disagree with them, has too much
>> the flavour of the a-priori dismissiveness of citation analysis
>> that we
>> hear in *all* disciplines from those who have not really investigated
>> it, or the evidence for/against it, but are simply expressing
>> their own
>> personal prejudices on the subject.
>>
>> Let's see the citation counts for humanities articles and books, and
>> their correlation with other performance indicators as well as RAE
>> rankings rather than dismissing them a-priori on the basis of
>> anecdotes.
>>
>>>   "Also, an analysis of RAE 2001 submissions revealed that while
>>> some
>>>   90% of research outputs listed by British researchers in the
>>> fields
>>>   of Physics and Chemistry were mapped by ISI data, in Law the
>>> figure
>>>   was below 10%, according to Ian Diamond of the Economic and Social
>>>   Research Council (ESRC) (Oxford Workshop on the use of Metrics in
>>>   Research Assessment)."
>>
>> I am not sure what "mapped by ISI data" means, but if it means
>> that ISI
>> does not
>> cover enough of the pertinent journals in Law, then the empirical
>> question
>> is:
>> what are the pertinent journals? can citation counts be derived
>> from their
>> online
>> versions, using the publishers' websites and/or subscribing
>> institutions'
>> online
>> versions? how well does this augmented citation count correlate
>> with the
>> ISI
>> subsample (<10%)? and how well do both correlate with RAE ranking?
>> (Surely
>> ISI
>> coverage should not be the determinant of whether or not a metric is
>> valid.)
>>
>>>   "Ultimately, a combination of qualitative and quantitative
>>> indicators
>>>   would seem to be the best approach."
>>
>> What is a "qualitative indicator"? A peer judgment of quality? But
>> that quality judgment has already been made by the peer-reviewers of
>> the journal in which the article was published -- and every field
>> has a
>> hierarchy of quality among journals that is known (and may even
>> sometimes
>> be correlated with the journal's impact factor, if one compares like
>> with like in terms of subject matter). What is the point of repeating
>> the peer review exercise? And especially if here too it turns out
>> to be
>> correlated with metrics? Is it?
>>
>>>   "While metrics are likely to be used to simplify the research
>>>   assessment process, the merits of a qualitative element would
>>> be to
>>>   ensure that over-reliance on quantitative factors does not
>>> unfairly
>>>   discriminate against research which is of good quality but has not
>>>   been cited as highly as other research due to factors such as its
>>>   local impact."
>>
>> Why not ask the panels first to make quality judgments on the
>> journals in
>> which
>> the papers were published, and then see whether those rankings
>> correlate
>> with the
>> author/article citation metrics? and whether they correlate with
>> the RAE
>> rankings
>> based on the present time-consuming qualitative re-evaluations? If
>> the
>> correlations prove lower than in the other fields (even when
>> augmented by
>> prior
>> funding and other metrics) *then* there may be a case for special
>> treatment of
>> the humanities. Otherwise, the special pleading on behalf of uncited
>> research
>> sounds as anecdotal, arbitrary and ad hoc as the claim that high
>> citations
>> in humanities betoken disagreement rather than usage and importance,
>> as in other fields.
>>
>>>   "Unless more appropriate metrics can be developed for the
>>> humanities,
>>>   it would seem that an element of expert peer review must remain in
>>>   whatever metrics based system emerges from the ashes of the RAE."
>>
>> It has not yet been shown whether the same metrics that correlate
>> highly
>> with RAE outcome in other fields (funding, citations) truly fail to
>> do so in the humanities. If they do fail to correlate sufficiently,
>> there are still many candidate metrics to try out (co-citations,
>> downloads, book citations, CiteRank, latency/longevity, exogamy/
>> endogamy,
>> hubs/authorities, co-text, etc.) before having to fall back on
>> repeating,
>> badly, the peer evaluation that should have already have been done,
>> properly, by the journals in which the research first appeared.
>>
>> Stevan Harnad
>>
>>> Research Assessment Exercise:  http://www.rae.co.uk
>>> Economic and Social Research Council:  http://www.eserc.ac.uk
>>>
>>> Open access: practical matters become the key focus, EPS
>>> Insights, 10
>>> March 2005
>>> http://www.epsltd.com/accessArticles.asp?
>>> articleType=1&updateNoteID=1538
>>>
>>> Citation Analysis in the Open Access World, imi, September 2004
>>> http://www.epsltd.com/accessArticles.asp?
>>> articleType=2&articleID=236&imiID=294

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20060409/08176560/attachment.html>