New citation indicators needed to measure research performance

[Loet Leydesdorff]

Dear Johannes,

In my opinion, one should distinguish between data collection and data
analysis. With Lutz Bornmann, I developed for example the program I3.exe
(available at http://www.leydesdorff.net/software/i3 ) that uses percentile
ranks on *any* dataset downloaded from WoS. The routine is rather similar
from that of Garfield and Pudovkin, but the user can freely choose the
reference set (journal or WoS category, or perhaps based on an informed
search string).

Best,
Loet

On Wed, Feb 20, 2013 at 10:36 PM, Johannes Stegmann <
johannes.stegmann at fu-berlin.de> wrote:

> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/**sigmetrics.html<http://web.utk.edu/~gwhitney/sigmetrics.html>
>
>  New citation indicators needed to measure research performance
>> HEIDELBERG, 12 February 2013 – How do you compare the impact of a
>> researcher in chemistry or physics with a molecular biologist who may be
>> working on similar projects? In an article published today in EMBO reports
>> two experts support the use of citation indicators that are based on
>> percentiles, a statistical parameter that allows for comparisons with a
>> carefully defined group of reference data. Journal impact factors and
>> h-index alone do not make the grade ….
>>
>> See http://www.embo.org/news/**research-news/new-citation-**
>> indicators-needed-to-measure-**research-performance<http://www.embo.org/news/research-news/new-citation-indicators-needed-to-measure-research-performance>
>>
>>
>
> Dear Sigmetricians,
>
> the above message by Lutz Bornmann points to his (and Werner Marx') paper
> "How good is research really? Measuring the citation impact of publications
> with percentiles increases correct assessments and fair comparisons", EMBO
> reports advance online publication 12 February 2013;
> doi:10.1038/embor.2013.9.
>
> I would like to draw your attention to the contribution of A. Pudovkin and
> E. Garfield to the COLLNET conference in Dalian, subsequently published in
> the Collnet Journal:
>
> Pudovkin, A.I. and Garfield, E. (2009): Percentile Rank and Author
> Superiority Indexes for Evaluating Individual Journal Articles and the
> Author’s Overall Citation Performance.
> Collnet Journal of Scientometrics and Information Management 3(2), 3–10
>
> The paper introduces the "Percentile Rank Index" (PRI) and the "Author’s
> Superiority Index" (ASI) derived from the PRI's of an author's papers.
> "The Percentile Rank Index (PRI) indicates the citation rank of the
> author’s individual papers among the papers published in the same year and
> source (journal or multi-authored monograph or book)." (cited from the
> abstract of Pudovkin & Garfield).
>
> In their EMBO-Reports paper, Bornmann & Marx argue for subject categories
> and against individual journals as reference set:
>
> "As an alternative to subject category, it is also possible to base the
> calculation of the expected citation impact on the journal in which a
> certain publication has appeared. However, individual journals are not an
> appropriate source from which to generate reference sets; manuscripts in
> high-impact journals, such as Science or Nature, would be penalized as the
> yard-stick would be higher. Conversely, publications in low-impact journals
> would seem to score highly, as it is easier to achieve a comparatively
> high-citation impact measured
> against a low-journal reference set." (cited from Bornmann & Marx).
>
> Why should a paper be "penalized" being published in a journal with an
> high impact factor? If one happens that his/her paper is accepted by the
> high-IF journal, why it should not be judged by the same criteria as the
> other papers published in the same journal in the same year? On the other
> hand, you could not seriously rank, e.g., a biophysics paper published in
> NATURE against the papers contained in the BIOPHYSICS-category because
> NATURE is not included in that category - or, vice versa, an arbitrary
> biophysics paper should be ranked against all papers in biophysics,
> including those published in multi-disciplinary journals.
> Pudovkin & Garfield make some helpful comments on PRI's of papers
> published in multi-disciplinary journals with respect to different citation
> behaviors in different subject fields which may be represented in one
> volume of, e.g., NATURE.
>
> And: "low-impact journals" means only low journal IF but not necessarily
> low value.
>
> The PRI based on individual journals as reference  may serve as an
> addition to other ranking mechanism (see Pudovkin & Garfield). Calculating
> the PRI (also) on the basis of whole subject categories would be certainly
> beneficial to an paper/author/institute evaluation. However, for "normal"
> people not having the whole WoS database under his/her desktop with
> unlimited access including the possibility to run programs/scripts directly
> on the WoS data, it is (nearly) impossible to get whole categories
> downloaded from the WoS because they may contain tens of thousands of
> papers within one single year. And WoS' "Citation Report" feature is
> available only for not more than 10,000 papers. A single journal, on the
> other hand, normally contains manageable number of papers within a year.
> Thus, if one has to deal with a handful of papers only, the PRI as defined
> by Pudovkin & Garfield is certainly more suited for assessment. If one has
> to deal with whole institutes or larger research entities, then one runs
> inevitably into problems also with the Pudovkin & Garfield method due to
> the big number of journal-year pairs one has to retrieve and either to
> download for offline processing and calculation of citation ranks or to use
> online the WoS Citation-Report feature.
>
> With kind regards,
> Johannes Stegmann
>
>
> --
> Dr. Johannes Stegmann
> Scientist
> Berlin, Germany
> email: johannes.stegmann at fu-berlin.de
>
>


-- 
Professor Loet Leydesdorff
Amsterdam School of Communications Research (ASCoR)
Kloveniersburgwal 48, 1012 CX Amsterdam
Tel.: +31-20- 525 6598; fax: +31-20- 525 3681
loet at leydesdorff.net ; http://www.leydesdorff.net/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20130221/3627134e/attachment.html>