Hefce backs off citations in favour of peer review in REF

[Stephen J Bensman]

In re your comment:

 

Moreover, it's not just about citations any more. A growing battery of research performance metrics need to be jointly validated and initialized against peer ranking. That's what the UK RAE/REF makes possible, uniquely, at a national, pandisciplinary scale

 

That is why I sent you the NRC URLs with their statistics.  You can take a look at a whole battery of different measures and correlate them with peer ratings.  Peer ratings-or, better stated, reputational ratings-are the oldest and still most important measure.  They were first proposed and justified by Francis Galton in his 1869 book Hereditary Genius.  The problem with peer ratings is that they are dependent upon the level of consensus in a given field, and, if a given field has no or low consensus, then you are nowhere.  Even your metrics are no good.  Under these conditions, if a government needs some type of measure to hand out money to universities, then a trip to the roulette table would be the fairest--and politically the safest-way.  

 

Stephen J. Bensman

LSU Libraries

Louisiana State University

Baton Rouge, LA   70803

USA

notsjb at lsu.edu

 

From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stevan Harnad
Sent: Wednesday, June 24, 2009 9:35 AM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] Hefce backs off citations in favour of peer review in REF

 

9:25 AM, Stephen J Bensman <notsjb at lsu.edu> wrote: 

	that is pretty much how Garfield recommended citations should be used and how they are used in US evaluations.  You don't use citations by themselves but to balance your subjective judgments.  

Gene is of course right that citations alone are not and never were enough for research evaluation; they not only need to be "balanced" against subjective (peer expert) evaluations, but they need to be formally validated against them, discipline by discipline.  

 

Moreover, it's not just about citations any more. A growing battery of research performance metrics need to be jointly validated and initialized against peer ranking. That's what the UK RAE/REF makes possible, uniquely, at a national, pandisciplinary scale. http://bit.ly/etLvL

 

Until the full-scale joint validation exercise is conducted and analyzed, discipline by discipline, no one can say what percentage of the variance in the peer rankings the metric battery can predict. If it's 20-40%, then metrics can only be advisory, merely supplementary adjuncts to the more expensive and time-consuming peer rankings; if it's 70-90%, then it's the peer rankings that are the supplementary adjuncts to the metrics.

 

Stevan Harnad

 

On Wed, Jun 24, 2009 at 9:25 AM, Stephen J Bensman <notsjb at lsu.edu> wrote:

	 

	I hate to say it, but that is pretty much how Garfield recommended citations should be used and how they are used in US evaluations.  You don't use citations by themselves but to balance your subjective judgments.  For Garfield's recommendations, see the two URL below:

	http://www.garfield.library.upenn.edu/essays/v6p354y1983.pdf

	 http://www.garfield.library.upenn.edu/essays/v6p363y1983.pdf

	For the most the most recent US National Research Council Data and methodology, see the following URLs:

	http://www.nap.edu/readingroom/books/researchdoc/

	http://sites.nationalacademies.org/pga/Resdoc/index.htm

	Given the politics of the thing, nobody in his right mind would use a purely metric approach, if he/she had any instinct for survival.    

	Stephen J. Bensman

	LSU Libraries

	Louisiana State University

	Baton Rouge, LA   70803

	USA

	notsjb at lsu.edu

	From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stevan Harnad
	Sent: Wednesday, June 24, 2009 3:26 AM
	To: SIGMETRICS at LISTSERV.UTK.EDU
	Subject: [SIGMETRICS] Hefce backs off citations in favour of peer review in REF 

	Hefce backs off citations in favour of peer review in REF

	18 June 2009

	By Zoë Corbyn <http://www.timeshighereducation.co.uk/biography.asp?contact=20> 

	Research assessments in hard sciences will now be 'informed' by bibliometrics. Zoe Corbyn writes

	The use of citations to determine the quality of academic work in the hard sciences is to be abandoned in favour of peer review in the new system being designed to replace the research assessment exercise.

	However, information about the number of citations a scholar's work accrues could be provided to assessment panels to help "inform" their judgments in a range of subjects....

	http://www.timeshighereducation.co.uk/story.asp?sectioncode=26&storycode=407041&c=1

	·         Richard Hull 20 June, 2009

	So finally common sense prevails. But I would now like to know exactly which stupid, thoughtless person, blinded by the New Labour mantra of "evidence-based this that and the other", first proposed the hair-brained idea to use citations?? Time for some journalistic digging, I think. This person must be exposed, as they have effectively wasted a huge amount of the time and energy of HEFCE and indeed the academics who actively opposed the idea.

	·         Stevan Harnad 22 June, 2009

	It's probably alright that instead of scrapping panel rankings altogether and hard-wiring the outcome to metrics, the new REF will continuing doing rankings and metrics in parallel, using the metrics as advisory rather than binding.  

	That's fine; it will give the metrics a better chance to be cross-validated against peer judgment (though the hybrid metric-influenced rankings of the new REF will not be as independent a criterion against which to validate metrics as the RAE rankings were, when they were not influenced by metrics). 

	The important thing is to make the battery of candidate metrics as broad and rich as possible. It is true that metrics today are still relatively sparse, but with the growth of open access and a rich variety of web-based metrics emerging therefrom, the power and scope of metrics will now grow and grow. 

	About the possibility of abuse: Yes, one can abuse individual metrics. Downloads are the easiest to abuse. But genuine downloads generate genuine citations, and the correlation is there and can be measured. There are other intercorrelations in multiple metric profiles too. There are endogamy/exogamy metrics: Self-citations, co-author citations, author-circle citations, same-institution citations, same-journal citations. With these, anomalies and abuses can be detected, named and shamed. 

	Multiple metrics create a pattern, a profile. If you artificially manipulate one of them (say, downloads, or citing others in your institution) it will be detectable as a deviation from the normal profile. Once a few of these abuses are prominently exposed and shamed, that will create a strong deterrent against trying such tricks, since the objective is the exact opposite: to increase one's prestige, not to tarnish it. 

	And unlike (some) individual metrics, multiple metric profiles are almost impossible to manipulate jointly: Try writing software to generate bogus downloads of your work looking as if they all come from different IPs the world over, and then try to generate the non-institutional citations that would normally be the correlate of such high downloads. Even that 2-metric trick is not easy to accomplish! 

	Stevan Harnad University of Southampton

	REPLY TO RICHARD HULL: ON EXPOSING THE CULPRIT -- Harnad, S. (2001) Research access, impact and assessment. Times Higher Education Supplement 1487: p. 16. http://cogprints.org/1683/

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20090624/566b371f/attachment.html>