"Bibliometric Distortion": The Babblarazzi Are At It Again...

[Jonathan Adams]

I recall the earlier debate.  I hear what you're saying, but disagree
with the disconnect you imply ('the model is not specified').
An expert group knows very well the relationship between measurement via
weighting to model.  Researchers are such an expert group, as are
financial market makers.
But your point on THES rankings is well taken.  Excellence does not come
cheap.  The amount that Harvard earned on its endowment fund last year
($5.7 Billion growth) is equivalent to 50 per cent of the entire HEFCE
grant to all English universities in the same period!
Sincere regards,

Jonathan Adams
 
Director, Evidence Ltd
+ 44 113 384 5680
 

-----Original Message-----
From: ASIS&T Special Interest Group on Metrics
[mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Loet Leydesdorff
Sent: 16 November 2007 14:01
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] "Bibliometric Distortion": The Babblarazzi Are
At It Again...


In my opinion, it is based on a methodological confusion which we
discussed with Steven Harnad previously on this list. The idea of a
metric is based on a multivariate model. The measurement then serves
the estimation of the parameters. The model guides the prediction.

In this case, the measurement is used as an independent predictor and
the model is not specified. The political system can then play with
the rankings which can vary according to the weights which one
attributes to the various parameters (e.g., in the normalization).

For example, the recent university rankings in the Times Higher
Education Supplement of last week. If one would propose to divide the
various scores by the budgets in order to estimate the efficiency of
output/input, the American and British universities which are now
leading the ranking would probably be at the bottom. :-)

Best wishes,


Loet

On 11/16/07, Jonathan Adams <Jonathan.adams at evidence.co.uk> wrote:
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Exactly so.  So long as the indicators remain separate from policy
> effects (such as funding) they remain sound.
> The problem arises where indicators become a target for the purpose of
> conducting policy, as Goddard (1975) suggested in regard to UK
> economics:
> http://en.wikipedia.org/wiki/Goodhart%27s_law
> And Campbell (1976) in relation to social science
> http://en.wikipedia.org/wiki/Campbell%27s_Law
>
> Jonathan Adams
>
> Director, Evidence Ltd
> + 44 113 384 5680
>
>
> -----Original Message-----
> From: ASIS&T Special Interest Group on Metrics
> [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Loet Leydesdorff
> Sent: 16 November 2007 12:41
> To: SIGMETRICS at listserv.utk.edu
> Subject: Re: [SIGMETRICS] "Bibliometric Distortion": The Babblarazzi
Are
> At It Again...
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> If I correctly remember, the Leiden normalization implies that one
> compares
> the citation scores with the expected citation scores given the
> publication
> profile of a group. You are right that a game follows naturally: if
one
> publishes in journals below one's level, one can expect to obtain a
> higher
> than expected citation score. Since all distributions are skewed, this
> effect would be reinforced.
>
> Hitherto, this has not been a major problem because the scores where
not
> directly related as input to funding.
>
> With best wishes,
>
>
> Loet
>
> > -----Original Message-----
> > From: ASIS&T Special Interest Group on Metrics
> > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Jonathan Adams
> > Sent: Friday, November 16, 2007 1:26 PM
> > To: SIGMETRICS at LISTSERV.UTK.EDU
> > Subject: Re: [SIGMETRICS] "Bibliometric Distortion": The
> > Babblarazzi Are At It Again...
> >
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> > I'm not planning this.  I understand the recommendation is from our
> > colleagues at Leiden, in a report to HEFCE that will be made public
> > later this month.
> > So far as outputs in Thomson-indexed journals go, I think
> > it's feasible
> > (and we have been analysing some scenarios using earlier data
> > reconciliation and analyses we did for HEFCE) but I wouldn't
recommend
> > it because of the games playing that I suspect would ensue.
> >
> > Jonathan Adams
> >
> > Director, Evidence Ltd
> > + 44 113 384 5680
> >
> >
> > -----Original Message-----
> > From: ASIS&T Special Interest Group on Metrics
> > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Loet Leydesdorff
> > Sent: 16 November 2007 12:05
> > To: SIGMETRICS at listserv.utk.edu
> > Subject: Re: [SIGMETRICS] "Bibliometric Distortion": The
> > Babblarazzi Are
> > At It Again...
> >
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> > > The proposal
> > > is that post-2008 the metrics assessment would be of all output,
> > > creating a profile and then deriving a metric derived from that.
> >
> > Dear Jonathan,
> >
> > How are you planning to do this? Interesting.
> >
> > Best wishes,
> >
> >
> > Loet
> >
>


-- 
Loet Leydesdorff
Amsterdam School of Communications Research (ASCoR)
Kloveniersburgwal 48, 1012 CX Amsterdam
Tel.: +31-20- 525 6598; fax: +31-20- 525 3681
loet at leydesdorff.net ; http://www.leydesdorff.net/
---------------------------------------
Now available: The Knowledge-Based Economy: Modeled, Measured,
Simulated, 385 pp.; US$ 18.95;