Continuous multi-metric research assessment

[Stevan Harnad]

There is no contradiction or conflict between generating open,  
continuous metrics and using them to measure and reward research  
performance, continuously. Yes, every formula can be abused. But  
abuses can be detected -- especially in the form of anomalous  
profiles within a multivariate formula. A univariate metric is far  
easier to abuse than a profile of inter-metric relations, both within  
a single author and across authors in a field.  Abuses can be  
penalized and formulas can be adjusted. And open scrutiny is itself a  
deterrent to cheating and manipulation, especially for academics. -- SH

On 16-Nov-07, at 7:40 AM, Loet Leydesdorff wrote:

> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> If I correctly remember, the Leiden normalization implies that one  
> compares
> the citation scores with the expected citation scores given the  
> publication
> profile of a group. You are right that a game follows naturally: if  
> one
> publishes in journals below one's level, one can expect to obtain a  
> higher
> than expected citation score. Since all distributions are skewed, this
> effect would be reinforced.
>
> Hitherto, this has not been a major problem because the scores  
> where not
> directly related as input to funding.
>
> With best wishes,
>
>
> Loet
>
>> -----Original Message-----
>> From: ASIS&T Special Interest Group on Metrics
>> [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Jonathan Adams
>> Sent: Friday, November 16, 2007 1:26 PM
>> To: SIGMETRICS at LISTSERV.UTK.EDU
>> Subject: Re: [SIGMETRICS] "Bibliometric Distortion": The
>> Babblarazzi Are At It Again...
>>
>> Adminstrative info for SIGMETRICS (for example unsubscribe):
>> http://web.utk.edu/~gwhitney/sigmetrics.html
>>
>> I'm not planning this.  I understand the recommendation is from our
>> colleagues at Leiden, in a report to HEFCE that will be made public
>> later this month.
>> So far as outputs in Thomson-indexed journals go, I think
>> it's feasible
>> (and we have been analysing some scenarios using earlier data
>> reconciliation and analyses we did for HEFCE) but I wouldn't  
>> recommend
>> it because of the games playing that I suspect would ensue.
>>
>> Jonathan Adams
>>
>> Director, Evidence Ltd
>> + 44 113 384 5680
>>
>>
>> -----Original Message-----
>> From: ASIS&T Special Interest Group on Metrics
>> [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Loet Leydesdorff
>> Sent: 16 November 2007 12:05
>> To: SIGMETRICS at listserv.utk.edu
>> Subject: Re: [SIGMETRICS] "Bibliometric Distortion": The
>> Babblarazzi Are
>> At It Again...
>>
>> Adminstrative info for SIGMETRICS (for example unsubscribe):
>> http://web.utk.edu/~gwhitney/sigmetrics.html
>>
>>> The proposal
>>> is that post-2008 the metrics assessment would be of all output,
>>> creating a profile and then deriving a metric derived from that.
>>
>> Dear Jonathan,
>>
>> How are you planning to do this? Interesting.
>>
>> Best wishes,
>>
>>
>> Loet
>>