New citation indicators needed to measure research performance

[Johannes Stegmann]

Dear Lutz, dear Loet.

@Lutz: Of course, I agree with you that whatever evaluation technique is 
applied, it should be transparent.
If journals are the reference one could add the information of their 
positions
within their subject category in the respective publication year. 
However, categories
themselves might be questionable in terms of compositions, coverage, 
subfields.
To use the papers of the same journal in which the paper to be compared 
has been published as reference, seems (to me) intuitively to be more 
logical.
In addition: if you analyse a research institute then you have normally 
to deal with a mixture of low and high value journals which makes the 
normalisation per se. It would be really interesting to find out whether 
Figure 1 (comparison of institute 1 and institute 2) in your EMBO-Report 
paper would look like more or less the same or quite different if you 
apply percentile ranking using the journals involved as reference base.

@Loet: Data collection is boring because for each paper you want to rank 
you have to download the respective journal/publication year pair. For 
our DRFZ study (see Pudovkin et al. and Kretschmer et al, Scientometrics 
Vol. 93 (2012), pp 3-16 and 17-30) we had to deal with over 300 papers 
published (by the DRFZ) in nearly 100 journals. To determine the PRI of 
each paper we had to download more than 260 journal/publication year 
pairs which serve as reference for the ranking. The analysis with 
respect to the determination of the position of each of the DRFZ papers 
within its journal/year reference was/is easy using PERL and R scripts 
for extraction of document id's and TIMES CITED numbers and appropriate 
matching.
This kind of analysis would be also no problem using whole subject 
categories as reference; here, the download is not possible for a normal 
user (as I stated already in my previous email).

With kind regards,

Johannes