New citation indicators needed to measure research performance

[Bornmann, Lutz]

Dear Johannes,

Thank you for your comments!

Concerning the use of single journals as reference sets I would like to point out that there is a main difference between single journals and subject categories as reference sets. Whereas the citation impact values for different subject categories (percentiles or means) ONLY reflect the different citation behaviours within different fields, reflect the values for different journals not only the different behaviours but also the different qualities (measured by citations). We can expect that certain journals publish (on average) papers with a higher quality than other journals. So, the citation impact value for a journal is also quality driven, but not the value for a field. This feature leads to a problem with single journals as reference sets. If one uses them as reference set one has to inform also whether the evaluated publication set was normalized on journals with a high or low average citation impact.

In general, reference sets should normalize for factors (influencing citation counts) without any relation to scientific quality.

Concerning the papers published in Nature or Science, I think we have to differentiate between journal sets as subject categories and categories where each single paper is assigned to a field. We agree that journal sets result in certainly not satisfactory normalized values in these cases. However, using for example the sections provided by Chemical Abstracts where papers are assigned to field categories on a paper-by-paper basis lead also for papers published in Science or Nature to proper results.

Best,

Lutz

Gesendet von Windows-Mail

Von: Johannes Stegmann
Gesendet: ‎20‎. ‎Februar‎ ‎2013 ‎22‎:‎47
An: SIGMETRICS at LISTSERV.UTK.EDU
Betreff: Re: [SIGMETRICS] New citation indicators needed to measure research performance

Adminstrative info for SIGMETRICS (for example unsubscribe):
http://web.utk.edu/~gwhitney/sigmetrics.html

> New citation indicators needed to measure research performance
> HEIDELBERG, 12 February 2013 – How do you compare the impact of a researcher in chemistry or physics with a molecular biologist who may be working on similar projects? In an article published today in EMBO reports two experts support the use of citation indicators that are based on percentiles, a statistical parameter that allows for comparisons with a carefully defined group of reference data. Journal impact factors and h-index alone do not make the grade ….
>
> See http://www.embo.org/news/research-news/new-citation-indicators-needed-to-measure-research-performance
>


Dear Sigmetricians,

the above message by Lutz Bornmann points to his (and Werner Marx')
paper "How good is research really? Measuring the citation impact of
publications with percentiles increases correct assessments and fair
comparisons", EMBO reports advance online publication 12 February 2013;
doi:10.1038/embor.2013.9.

I would like to draw your attention to the contribution of A. Pudovkin
and E. Garfield to the COLLNET conference in Dalian, subsequently
published in the Collnet Journal:

Pudovkin, A.I. and Garfield, E. (2009): Percentile Rank and Author
Superiority Indexes for Evaluating Individual Journal Articles and the
Author’s Overall Citation Performance.
Collnet Journal of Scientometrics and Information Management 3(2), 3–10

The paper introduces the "Percentile Rank Index" (PRI) and the "Author’s
Superiority Index" (ASI) derived from the PRI's of an author's papers.
"The Percentile Rank Index (PRI) indicates the citation rank of the
author’s individual papers among the papers published in the same year
and source (journal or multi-authored monograph or book)." (cited from
the abstract of Pudovkin & Garfield).

In their EMBO-Reports paper, Bornmann & Marx argue for subject
categories and against individual journals as reference set:

"As an alternative to subject category, it is also possible to base the
calculation of the expected citation impact on the journal in which a
certain publication has appeared. However, individual journals are not
an appropriate source from which to generate reference sets; manuscripts
in high-impact journals, such as Science or Nature, would be penalized
as the yard-stick would be higher. Conversely, publications in
low-impact journals would seem to score highly, as it is easier to
achieve a comparatively high-citation impact measured
against a low-journal reference set." (cited from Bornmann & Marx).

Why should a paper be "penalized" being published in a journal with an
high impact factor? If one happens that his/her paper is accepted by the
high-IF journal, why it should not be judged by the same criteria as the
other papers published in the same journal in the same year? On the
other hand, you could not seriously rank, e.g., a biophysics paper
published in NATURE against the papers contained in the
BIOPHYSICS-category because NATURE is not included in that category -
or, vice versa, an arbitrary biophysics paper should be ranked against
all papers in biophysics, including those published in
multi-disciplinary journals.
Pudovkin & Garfield make some helpful comments on PRI's of papers
published in multi-disciplinary journals with respect to different
citation behaviors in different subject fields which may be represented
in one volume of, e.g., NATURE.

And: "low-impact journals" means only low journal IF but not necessarily
low value.

The PRI based on individual journals as reference  may serve as an
addition to other ranking mechanism (see Pudovkin & Garfield).
Calculating the PRI (also) on the basis of whole subject categories
would be certainly beneficial to an paper/author/institute evaluation.
However, for "normal" people not having the whole WoS database under
his/her desktop with unlimited access including the possibility to run
programs/scripts directly on the WoS data, it is (nearly) impossible to
get whole categories downloaded from the WoS because they may contain
tens of thousands of papers within one single year. And WoS' "Citation
Report" feature is available only for not more than 10,000 papers. A
single journal, on the other hand, normally contains manageable number
of papers within a year. Thus, if one has to deal with a handful of
papers only, the PRI as defined by Pudovkin & Garfield is certainly more
suited for assessment. If one has to deal with whole institutes or
larger research entities, then one runs inevitably into problems also
with the Pudovkin & Garfield method due to the big number of
journal-year pairs one has to retrieve and either to download for
offline processing and calculation of citation ranks or to use online
the WoS Citation-Report feature.

With kind regards,
Johannes Stegmann


--
Dr. Johannes Stegmann
Scientist
Berlin, Germany
email: johannes.stegmann at fu-berlin.de

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20130221/f04cda6c/attachment.html>