Caveats for the journal and field normalizations in the CWTS ("Leiden") evaluations of research performance

[Linda Butler]

Loet, Ludo

There is another reason why bibliometricians are starting to do  
calculations at the individual level, which doesn't seem to have been  
mentioned in any of the papers referred to in this discussion.

That reason is to make it easier to re-aggregate publication sets in  
different ways.

Let me give a "live" example.  The Australian Research Council (ARC)  
trialled a new assessment system late last year.  In physics,  
chemistry and earth sciences (the only sciences assessed in the trial  
- the other disciplines were all in the humanities and creative arts)  
bibliometrics played a key role.  Several indicators were used, of  
which one was Relative Citation Impact.  It was calculated by  
determining the RCI for each individual paper, then computing an  
average of RCIs to give an institutional RCI i.e. the method Loet is  
championing and the one that Ludo reveals CWTS is now also working on.

In addition to statistical considerations, there was an additional  
important advantage to the ARC in using this methodology.  The  
Australian Government was only interested in assessing the  
performance of fields within institutions - who had the strongest/ 
weakest geology, astronomy, organic chemistry, etc.  They were not  
concerned about which academic units these publications came from.   
But Vice-Chancellors and research managers were!  VERY interested!!   
 From the very first discussions on the development of the  
methodology (late 2007) it became clear that an essential  
consideration was the ability of universities to re-aggregate data to  
groups, faculties, research themes, or however they wanted to do.  I  
am not necessarily supporting what they want to do - but they  
undoubtedly want to do it.  An article by article methodology makes  
this a simple computational task.

So the methodology has already being used in a national research  
assessment system, and will be used again when all fields that are  
subject to bibliometrics are assessed in the second half of this year.

Can I just add that I'm just a little uncomfortable that CWTS seems  
to have been singled out for such pointed criticism.  It's not just a  
"CWTS" methodology.  Many other groups have used or are using this  
method, and not just because they copied CWTS - my own unit, REPP, is  
a case in point.  Many of us arrived at the same point through our  
own development work, and having arrived there, are now looking to  
move on and improve the calculations.

It's an important issue, so let's keep the discussion going, but keep  
it collegial.

regards
Linda Butler


On 20/03/2010, at 9:22 AM, Loet Leydesdorff wrote:

> Adminstrative info for SIGMETRICS (for example unsubscribe): http:// 
> web.utk.edu/~gwhitney/sigmetrics.html
> Dear colleagues,
>
> Please, find below the CWTS indicators for four papers of which two  
> are cited in the same journal i or the same field i:
>
> <Outlook.jpg>
> <Outlook.jpg>
> In our opinion, one should normalize as follows:
>
> <Outlook.jpg>
>
> The issue is more general than CWTS because other centers normalize  
> using MOCR/MECR, that is: Mean Observed Citation Rates divided by  
> Mean Expected Citation Rates. The quotient between two means is no  
> longer a statistics, while the values of observed versus expected  
> citation rates provide a variance (standard deviation, median, etc.).
> (For example, 3/2 plus 2/3 is very different from 5/5).
>
> We understand from the papers indicated that in the meantime CWTS  
> is changing its procedures.
>
> Best wishes,
>
> Loet
> ________________________________
>
> Loet Leydesdorff
> Amsterdam School of Communications Research (ASCoR),
> Kloveniersburgwal 48, 1012 CX Amsterdam.
> Tel.: +31-20- 525 6598; fax: +31-20- 525 3681
> loet at leydesdorff.net ; http://www.leydesdorff.net/
>
>
>
> > -----Original Message-----
> > From: ASIS&T Special Interest Group on Metrics
> > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Ludo Waltman
> > Sent: Friday, March 19, 2010 10:34 PM
> > To: SIGMETRICS at LISTSERV.UTK.EDU
> > Subject: Re: [SIGMETRICS] Caveats for the journal and field
> > normalizations in the CWTS ("Leiden") evaluations of research
> > performance
> >
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> > Dear colleagues,
> >
> > Recently, Tobias Opthof and Loet Leydesdorff have written a
> > critical paper
> > (see below) about the way in which bibliometric research performance
> > assessment studies are conducted by the Centre for Science
> > and Technology
> > Studies (CWTS) of Leiden University. There are a number of important
> > inaccuracies in the paper by Opthof and Leydesdorff. CWTS
> > also strongly
> > disagrees with many of their comments. In the following paper
> > CWTS replies to
> > the criticism of Opthof and Leydesdorff:
> >
> > Anthony F.J. van Raan, Thed N. van Leeuwen, Martijn S.
> > Visser, Nees Jan van
> > Eck, and Ludo Waltman. Rivals for the crown: Reply to Opthof and
> > Leydesdorff. Available at http://arxiv.org/abs/1003.2113.
> >
> > CWTS has also prepared a related paper on the same topic:
> >
> > Ludo Waltman, Nees Jan van Eck, Thed N. van Leeuwen, Martijn
> > S. Visser, and
> > Anthony F.J. van Raan. Towards a new crown indicator: Some
> > theoretical
> > considerations. Available at http://arxiv.org/abs/1003.2167.
> >
> > Best regards,
> >
> > Ludo Waltman
> >
> >
> > On 16/02/2010 07:46, Loet Leydesdorff wrote:
> > > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> > > Caveats for the journal and field normalizations in the
> > CWTS ("Leiden")
> > evaluations of research performance
> > > Journal of Informetrics (forthcoming).
> > > http://arxiv.org/abs/1002.2769
> > >
> > >     Abstract: The Center for Science and Technology Studies
> > at Leiden
> > University advocates the use of specific normalizations for
> > assessing research
> > performance with reference to a world average. The Journal
> > Citation Score
> > (JCS) and Field Citation Score (FCS) are averaged for the
> > research group or
> > individual researcher under study, and then these values are used as
> > denominators of the (mean) Citations per publication (CPP).
> > Thus, this
> > normalization is based on dividing two averages. This procedure only
> > generates a legitimate indicator in the case of underlying
> > normal distributions.
> > Given the skewed distributions under study, one should
> > average the observed
> > versus expected values which are to be divided first for each
> > publication. We
> > show the effects of the Leiden normalization for a recent
> > evaluation where we
> > happened to have access to the underlying data.
> > >
> > >
> > > Tobias Opthof [1,2], Loet Leydesdorff [3]
> > >
> > >
> > > [1] Experimental Cardiology Group, Heart Failure Research
> > Center, Academic
> > Medical Center AMC, Meibergdreef 9, 1105 AZ Amsterdam, The
> > Netherlands.
> > >
> > > [2] Department of Medical Physiology, University Medical
> > Center Utrecht,
> > Utrecht, The Netherlands.
> > >
> > > [3] Amsterdam School of Communications Research (ASCoR),
> > University of
> > Amsterdam, Kloveniersburgwal 48, 1012 CX Amsterdam, The Netherlands.
> > >
> > >
> > >
> > > ** apologies for cross-postings
> > >
> > >
> >
> >
> > ========================================================
> > Ludo Waltman MSc
> > Researcher
> >
> > Centre for Science and Technology Studies
> > Leiden University
> > P.O. Box 905
> > 2300 AX Leiden
> > The Netherlands
> >
> > Willem Einthoven Building, Room B5-35
> > Tel:      +31 (0)71 527 5806
> > Fax:      +31 (0)71 527 3911
> > E-mail:   waltmanlr at cwts.leidenuniv.nl
> > Homepage: www.ludowaltman.nl
> > ========================================================
> >

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Linda Butler
linda.butler52 at gmail.com

landline: +61 (0)2 4982 7994
mobile: 0428 598 482
url: http://members.optuszoo.com.au/linda.butler52
ABN:  83 884 783 826



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20100320/2c8c8a89/attachment.html>