Caveats for the journal and field normalizations in the CWTS ("Leiden") evaluations of research performance

[Loet Leydesdorff]

Dear colleagues,
 
Please, find below the CWTS indicators for four papers of which two are
cited in the same journal i or the same field i:
 


In our opinion, one should normalize as follows:
 


The issue is more general than CWTS because other centers normalize using
MOCR/MECR, that is: Mean Observed Citation Rates divided by Mean Expected
Citation Rates. The quotient between two means is no longer a statistics,
while the values of observed versus expected citation rates provide a
variance (standard deviation, median, etc.). 
(For example, 3/2 plus 2/3 is very different from 5/5).
 
We understand from the papers indicated that in the meantime CWTS is
changing its procedures.
 
Best wishes, 
 
Loet
________________________________

Loet Leydesdorff
Amsterdam School of Communications Research (ASCoR),
Kloveniersburgwal 48, 1012 CX Amsterdam.
Tel.: +31-20- 525 6598; fax: +31-20- 525 3681
loet at leydesdorff.net ; http://www.leydesdorff.net/



> -----Original Message-----
> From: ASIS&T Special Interest Group on Metrics
> [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Ludo Waltman
> Sent: Friday, March 19, 2010 10:34 PM
> To: SIGMETRICS at LISTSERV.UTK.EDU
> Subject: Re: [SIGMETRICS] Caveats for the journal and field
> normalizations in the CWTS ("Leiden") evaluations of research
> performance
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Dear colleagues,
>
> Recently, Tobias Opthof and Loet Leydesdorff have written a
> critical paper
> (see below) about the way in which bibliometric research performance
> assessment studies are conducted by the Centre for Science
> and Technology
> Studies (CWTS) of Leiden University. There are a number of important
> inaccuracies in the paper by Opthof and Leydesdorff. CWTS
> also strongly
> disagrees with many of their comments. In the following paper
> CWTS replies to
> the criticism of Opthof and Leydesdorff:
>
> Anthony F.J. van Raan, Thed N. van Leeuwen, Martijn S.
> Visser, Nees Jan van
> Eck, and Ludo Waltman. Rivals for the crown: Reply to Opthof and
> Leydesdorff. Available at http://arxiv.org/abs/1003.2113.
>
> CWTS has also prepared a related paper on the same topic:
>
> Ludo Waltman, Nees Jan van Eck, Thed N. van Leeuwen, Martijn
> S. Visser, and
> Anthony F.J. van Raan. Towards a new crown indicator: Some
> theoretical
> considerations. Available at http://arxiv.org/abs/1003.2167.
>
> Best regards,
>
> Ludo Waltman
>
>
> On 16/02/2010 07:46, Loet Leydesdorff wrote:
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
> > Caveats for the journal and field normalizations in the
> CWTS ("Leiden")
> evaluations of research performance
> > Journal of Informetrics (forthcoming).
> > http://arxiv.org/abs/1002.2769
> >
> >     Abstract: The Center for Science and Technology Studies
> at Leiden
> University advocates the use of specific normalizations for
> assessing research
> performance with reference to a world average. The Journal
> Citation Score
> (JCS) and Field Citation Score (FCS) are averaged for the
> research group or
> individual researcher under study, and then these values are used as
> denominators of the (mean) Citations per publication (CPP).
> Thus, this
> normalization is based on dividing two averages. This procedure only
> generates a legitimate indicator in the case of underlying
> normal distributions.
> Given the skewed distributions under study, one should
> average the observed
> versus expected values which are to be divided first for each
> publication. We
> show the effects of the Leiden normalization for a recent
> evaluation where we
> happened to have access to the underlying data.
> >     
> >
> > Tobias Opthof [1,2], Loet Leydesdorff [3]
> > 
> >
> > [1] Experimental Cardiology Group, Heart Failure Research
> Center, Academic
> Medical Center AMC, Meibergdreef 9, 1105 AZ Amsterdam, The
> Netherlands.
> >
> > [2] Department of Medical Physiology, University Medical
> Center Utrecht,
> Utrecht, The Netherlands.
> >
> > [3] Amsterdam School of Communications Research (ASCoR),
> University of
> Amsterdam, Kloveniersburgwal 48, 1012 CX Amsterdam, The Netherlands.
> >
> > 
> >
> > ** apologies for cross-postings
> >
> >
>
>
> ========================================================
> Ludo Waltman MSc
> Researcher
>
> Centre for Science and Technology Studies
> Leiden University
> P.O. Box 905
> 2300 AX Leiden
> The Netherlands
>
> Willem Einthoven Building, Room B5-35
> Tel:      +31 (0)71 527 5806
> Fax:      +31 (0)71 527 3911
> E-mail:   waltmanlr at cwts.leidenuniv.nl
> Homepage: www.ludowaltman.nl
> ========================================================
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20100319/f4ae70e6/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Outlook.jpg
Type: image/jpeg
Size: 14427 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20100319/f4ae70e6/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Outlook.jpg
Type: image/jpeg
Size: 13825 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20100319/f4ae70e6/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Outlook.jpg
Type: image/jpeg
Size: 15107 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20100319/f4ae70e6/attachment-0002.jpg>