New citation indicators needed to measure research performance

[Johannes Stegmann]

> New citation indicators needed to measure research performance
> HEIDELBERG, 12 February 2013 – How do you compare the impact of a researcher in chemistry or physics with a molecular biologist who may be working on similar projects? In an article published today in EMBO reports two experts support the use of citation indicators that are based on percentiles, a statistical parameter that allows for comparisons with a carefully defined group of reference data. Journal impact factors and h-index alone do not make the grade ….
>
> See http://www.embo.org/news/research-news/new-citation-indicators-needed-to-measure-research-performance
>


Dear Sigmetricians,

the above message by Lutz Bornmann points to his (and Werner Marx') 
paper "How good is research really? Measuring the citation impact of 
publications with percentiles increases correct assessments and fair 
comparisons", EMBO reports advance online publication 12 February 2013;
doi:10.1038/embor.2013.9.

I would like to draw your attention to the contribution of A. Pudovkin 
and E. Garfield to the COLLNET conference in Dalian, subsequently 
published in the Collnet Journal:

Pudovkin, A.I. and Garfield, E. (2009): Percentile Rank and Author 
Superiority Indexes for Evaluating Individual Journal Articles and the 
Author’s Overall Citation Performance.
Collnet Journal of Scientometrics and Information Management 3(2), 3–10

The paper introduces the "Percentile Rank Index" (PRI) and the "Author’s 
Superiority Index" (ASI) derived from the PRI's of an author's papers.
"The Percentile Rank Index (PRI) indicates the citation rank of the 
author’s individual papers among the papers published in the same year 
and source (journal or multi-authored monograph or book)." (cited from 
the abstract of Pudovkin & Garfield).

In their EMBO-Reports paper, Bornmann & Marx argue for subject 
categories and against individual journals as reference set:

"As an alternative to subject category, it is also possible to base the 
calculation of the expected citation impact on the journal in which a 
certain publication has appeared. However, individual journals are not 
an appropriate source from which to generate reference sets; manuscripts 
in high-impact journals, such as Science or Nature, would be penalized 
as the yard-stick would be higher. Conversely, publications in 
low-impact journals would seem to score highly, as it is easier to 
achieve a comparatively high-citation impact measured
against a low-journal reference set." (cited from Bornmann & Marx).

Why should a paper be "penalized" being published in a journal with an 
high impact factor? If one happens that his/her paper is accepted by the 
high-IF journal, why it should not be judged by the same criteria as the 
other papers published in the same journal in the same year? On the 
other hand, you could not seriously rank, e.g., a biophysics paper 
published in NATURE against the papers contained in the 
BIOPHYSICS-category because NATURE is not included in that category - 
or, vice versa, an arbitrary biophysics paper should be ranked against 
all papers in biophysics, including those published in 
multi-disciplinary journals.
Pudovkin & Garfield make some helpful comments on PRI's of papers 
published in multi-disciplinary journals with respect to different 
citation behaviors in different subject fields which may be represented 
in one volume of, e.g., NATURE.

And: "low-impact journals" means only low journal IF but not necessarily 
low value.

The PRI based on individual journals as reference  may serve as an 
addition to other ranking mechanism (see Pudovkin & Garfield). 
Calculating the PRI (also) on the basis of whole subject categories 
would be certainly beneficial to an paper/author/institute evaluation. 
However, for "normal" people not having the whole WoS database under 
his/her desktop with unlimited access including the possibility to run 
programs/scripts directly on the WoS data, it is (nearly) impossible to 
get whole categories downloaded from the WoS because they may contain 
tens of thousands of papers within one single year. And WoS' "Citation 
Report" feature is available only for not more than 10,000 papers. A 
single journal, on the other hand, normally contains manageable number 
of papers within a year. Thus, if one has to deal with a handful of 
papers only, the PRI as defined by Pudovkin & Garfield is certainly more 
suited for assessment. If one has to deal with whole institutes or 
larger research entities, then one runs inevitably into problems also 
with the Pudovkin & Garfield method due to the big number of 
journal-year pairs one has to retrieve and either to download for 
offline processing and calculation of citation ranks or to use online 
the WoS Citation-Report feature.

With kind regards,
Johannes Stegmann


-- 
Dr. Johannes Stegmann
Scientist
Berlin, Germany
email: johannes.stegmann at fu-berlin.de