Caveats for the journal and field normalizations in the CWTS ("Leiden") evaluations of research performance

Sat Apr 10 02:21:08 EDT 2010

Dear Fran and colleagues, 

Perhaps, this historical positioning in the 1980s is incorrect. The Leiden
("crown") indicators were published only in 1995 (Moed et al., 1995,
Scientometrics). Before that time, the Leiden colleagues only used
(unnormalized) descriptive statistics. The normalization problems should
have been noted by the referees at the time. However, the Budapest group had
developed during the 1980s -- in relative isolation -- the practice of
dividing the Mean Observed Citation Rate (MOCR) by the Mean Expected
Citation Rate (MECR) instead of taking the mean of the observed versus the
expected citation rates. This measure -- now called "normalized mean
citation rate" -- is still used by ECOOM in Leuven (Glaenzel et al., 2009,
Scientometrics). 

In summary, the difference will in the future be between the "mean
normalized citation score" (MNCS) to be used by Leiden as the new crown
indicator and the "normalized mean citation rate" (NMCR) used by Leuven. It
seems to me that this mixing of terminologies may easily confuse end users
and therefore not contribute to the transparency of using scientometric
indicators. The two indicators are conceptually rather different.

Best wishes, 

Loet

  _____  

Loet Leydesdorff 
Amsterdam School of Communications Research (ASCoR)
Kloveniersburgwal 48, 1012 CX Amsterdam
 <mailto:loet at leydesdorff.net> loet at leydesdorff.net ;
<http://www.leydesdorff.net/> http://www.leydesdorff.net/ 

  _____  

From: ASIS&T Special Interest Group on Metrics
[mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Francis Narin
Sent: Friday, April 09, 2010 10:24 PM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] Caveats for the journal and field normalizations
in the CWTS ("Leiden") evaluations of research performance

Dear Ulf,

    Loet's recollection is correct. I remember the car ride quite clearly.
In addition I did interact a lot with Van Raan and others in the group at
Leiden in the 1980's and 1990's , and also with the groups at SPRU, and in
London, Paris, Bielefeld and elsewhere in Europe.

        Francis Narin

----- Original Message ----- 
From: Loet  <mailto:loet at LEYDESDORFF.NET> Leydesdorff 
To: SIGMETRICS at LISTSERV.UTK.EDU 
Sent: Friday, April 09, 2010 12:03 PM
Subject: Re: [SIGMETRICS] Caveats for the journal and field normalizations
in the CWTS ("Leiden") evaluations of research performance

It was published just before the groups in Leiden and Leuven came with their
first articles on normalized citation analysis, but had quite a different,
and methodologically sound approach. In 2006, I visited Barry Bozeman in the
US and gave a presentation on field normalized citation analysis.
Immediately, he responded in a way that pointed to z-score as the best
alternative for handling these types of problems. Could it be the case that
in the 1980s there was a gap between the US and Europe, maybe there still
is, regarding statistical methods and methdological competence? 

Dear Ulf, 

Thank you for briging this paper to our attention. In the early 1980s there
was a lifely relation between the US and European scholars in bibliometrics.
We met, for example, at the meetings of the 4S (The Society for the Social
Studies of Science). I remember, for example, a car ride from a meeting in
Troy (NY) with Fran Narin as the driver and Henk Moed and me in the car.
Fran was going to show us his company in Philadelphia, but I felt not well
and stepped out of the car in Newark. 

Thus, we knew one another quite well. And certainly, we all read the same
journals. (I must confess that I had never read this excellent piece).

Best wishes, 

Loet

Since 2005, my group have performed citation analysis with normalization at
article level, and in 2007 we included the Standard Citation Score using the
method proposed by Mcallister et al. Most or our work is in Swedish, but to
give a couple of examples (in English) of this type of work: see the report
on the Research Assessement in 2008 for Royal Institute of Technology,
Stockholm, avaiable here:
http://www.forskningspolitik.se/DataFile.asp?FileID=165 or via www.se/rae.
Another evaluation using the same methods was performed last year for the
Swedish Environmental Protection Agency "Bibliometric evaluation of research
programs": http://www.forskningspolitik.se/DataFile.asp?FileID=182.
Especially, have a look at the Technical Appendices to these reports.

Best,
Ulf Sandstrom

Ulf Sandström, docent 

Linköpings universitet                 

ISAK                                        

581 83 Linköping                        

(           +46 708 137376
*           ulf.sandstrom at liu.se 
"**** www.forskningspolitik.se <http://www.forskningspolitik.se/>   

  _____  

Från: Loet Leydesdorff [mailto:loet at LEYDESDORFF.NET] 
Skickat: den 23 mars 2010 07:41
Ämne: Re: Caveats for the journal and field normalizations in the CWTS
("Leiden") evaluations of research performance

Normalization, CWTS indicators, and the Leiden Rankings: 
Differences in citation behavior at the level of fields

Authors: Loet
<http://arxiv.org/find/physics/1/au:+Leydesdorff_L/0/1/0/all/0/1>
Leydesdorff, Tobias
<http://arxiv.org/find/physics/1/au:+Opthof_T/0/1/0/all/0/1> Opthof
(Submitted on 21 Mar 2010)

Abstract: Van Raan et al. (2010; arXiv:1003.2113
<http://arxiv.org/abs/1003.2113> ) have proposed a new indicator (MNCS) for
field normalization. Since field normalization is also used in the Leiden
Rankings of universities, we elaborate our critique of journal normalization
in Opthof & Leydesdorff (2010; arXiv:1002.2769
<http://arxiv.org/abs/1002.2769> ) in this rejoinder concerning field
normalization. Fractional citation counting thoroughly solves the issue of
normalization for differences in citation behavior among fields. This
indicator can also be used to obtain a normalized impact factor. 

Subjects: 	Physics and Society (physics.soc-ph)	
Cite as: 	arXiv:1003.3977v1 <http://arxiv.org/abs/1003.3977v1>
[physics.soc-ph]	

  _____  

Loet Leydesdorff 
Amsterdam School of Communications Research (ASCoR)
Kloveniersburgwal 48, 1012 CX Amsterdam
 <mailto:loet at leydesdorff.net> loet at leydesdorff.net ;
<http://www.leydesdorff.net/> http://www.leydesdorff.net/ 

  _____  

No virus found in this incoming message.
Checked by AVG - www.avg.com 
Version: 9.0.801 / Virus Database: 271.1.1/2800 - Release Date: 04/08/10
23:32:00

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20100410/18e3eb53/attachment.html>