Percentile ranks in research evaluation

[Loet Leydesdorff]

The validation of (advanced) bibliometric indicators through peer
assessments: A comparative study using data from InCites and F1000
<http://arxiv.org/abs/1211.1154> 


The data of F1000 provide us with the unique opportunity to investigate the
relationship between peers' ratings and bibliometric metrics on a broad and
comprehensive data set with high-quality ratings. F1000 is a
post-publication peer review system of the biomedical literature. The
comparison of metrics with peer evaluation has been widely acknowledged as a
way of validating metrics. Based on the seven indicators offered by InCites,
we analyzed the validity of raw citation counts (Times Cited, 2nd Generation
Citations, and 2nd Generation Citations per Citing Document), normalized
indicators (Journal Actual/Expected Citations, Category Actual/Expected
Citations, and Percentile in Subject Area), and a journal based indicator
(Journal Impact Factor). The data set consists of 125 papers published in
2008 and belonging to the subject category cell biology or immunology. As
the results show, Percentile in Subject Area achieves the highest
correlation with F1000 ratings; we can assert that for further three other
indicators (Times Cited, 2nd Generation Citations, and Category
Actual/Expected Citations) the 'true' correlation with the ratings reaches
at least a medium effect size

 

Lutz Bornmann & Loet Leydesdorff

(Submitted on 6 Nov 2012)

  _____  


The use of percentiles and percentile rank classes in the analysis of
bibliometric data: Opportunities and limits <http://arxiv.org/abs/1211.0381>
, Journal of Informetrics (in press).


Percentiles have been established in bibliometrics as an important
alternative to mean-based indicators for obtaining a normalized citation
impact of publications. Percentiles have a number of advantages over
standard bibliometric indicators used frequently: for example, their
calculation is not based on the arithmetic mean which should not be used for
skewed bibliometric data. This study describes the opportunities and limits
and the advantages and disadvantages of using percentiles in bibliometrics.
We also address problems in the calculation of percentiles and percentile
rank classes for which there is not (yet) a satisfactory solution. It will
be hard to compare the results of different percentile-based studies with
each other unless it is clear that the studies were done with the same
choices for percentile calculation and rank assignment. 

 

Lutz Bornmann, Loet Leydesdorff, & Ruediger Mutz

  _____  

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20121107/9982d09a/attachment.html>