New Letter to the Editor

Mon Mar 30 08:51:41 EDT 2015

I agree with Lutz's general point, that any reasonable methodology will
usually produce similar results especially when applied to a reasonably
large sample of reasonably balanced data. We all recognise that the
underlying driver is that some teams/institutions/countries produce greater
numbers of more frequently cited publications. You have to be perverse for
them' not to 'do well'.

The problem that Loet is pointing us towards is that many analysts are
applying methodology towards smaller samples, or less well-balanced data,
and that they are teasing out factors regarding less 'peak' and more
'platform' performance. If they are delivering reports to a client or an
employing organisation then the methodology (and interpretation) they use
may have a significant and not always well-founded influence.

Jonathan Adams
Digital Science

On 30 March 2015 at 13:27, Bornmann, Lutz <lutz.bornmann at gv.mpg.de> wrote:

>  Hi Loet,
>
>
>
> I agree that we have very good alternatives for the MNCS and the use of
> WoS categories. However, the alternatives have their own (mostly practical)
> weaknesses. Furthermore, it seems that the different normalization methods
> produce similar results (see
> http://www.sciencedirect.com/science/article/pii/S1751157715000073).
>
>
>
> Perhaps, other people on this list can report which normalization method
> they use (as standard). In my opinion, it would be interesting to know this.
>
>
>
> Best,
>
>
>
> Lutz
>
>
>
> *From:* ASIS&T Special Interest Group on Metrics [mailto:
> SIGMETRICS at LISTSERV.UTK.EDU] *On Behalf Of *Loet Leydesdorff
> *Sent:* Monday, March 30, 2015 9:17 AM
> *To:* SIGMETRICS at LISTSERV.UTK.EDU
> *Subject:* Re: [SIGMETRICS] New Letter to the Editor
>
>
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> PS.
>
>
>
> Both discussions – the one about using the mean (MNCS) and the one about
> using WoS Subject Categories for the normalization – seem now to have
> stagnated.
>
>
>
> 1.       Instead of the mean, one should use percentile rank classes.
> This was a step in a line of thought in 2010-2011 in which we first
> criticized the “old” crown indicator and then proposed what later became
> labeled by CWTS as MNCS (Opthof & Leydesdorff, 2010; cf. Lundberg, 2007;
> Waltman et al., 2011). We subsequently moved to percentiles, and automated
> the “Integrated Impact Indicator” that enables users to define one’s
> percentile rank classes at http://www.leydesdorff.net/software/i3
> (Leydesdorff & Bornmann, 2011a).
>
>
>
> Another line of thought was source-normalization or fractional counting of
> the citations (Zitt & Small, 2008; Moed, 2010; Leydesdorff & Bornmann,
> 2011b). This was elaborated into the SNIP and then into SNIP2. I mentioned
> Mingers (2014) because this development seems to have got stuck now; the
> critique does no longer matter?) SJR-2 (Guerrero-Bote et al., 2012), of
> course, provides an alternative, but nobody can use this indicator outside
> the institute that constructed it.
>
>
>
> In my opinion, I3 and source-normalization (fractional counting) of the
> citations are still good ideas if one does not have WoS in-house through a
> license. Perhaps, this is an argument for what you call
> “amateur-bibliometrics”. It is better than taking the mean.
>
>
>
> 2.       In principle, SNIP and fractional counting creatively solve the
> determination of reference sets. The issue is not “normalization” per se,
> but the specification of an expectation (to be used in the denominator).
> The institutionalization in Scopus, however, may have been premature; or is
> there room to move to SNIP-3, and so forth? (Waltman et al., 2013). SNIP
> may be too technical to be reproduced (or controlled) outside the context
> of its production.
>
>
>
> The determination of reference sets in terms of journals may not work or
> not be possible (Rafols & Leydesdorff, 2009). The sets are fuzzy and remain
> changing. CWTS now moved in the Leiden Rankings 2014 to direct clustering
> of the citations, but the 800+ fields can no longer be validated
> (Ruiz-Castillo & Waltman, 2015). A disadvantage is that nobody can
> reproduce the results outside the institute which constructed these
> “fields”. We know that algorithmic constructs do not necessarily match with
> intellectual classifications. Furthermore, because the delineation is
> paper-based (instead of journal-based), one would have to update
> continuously. Thus, the “fields” cannot be reproduced at a next moment of
> time.
>
>
>
> If one is not able to specify an expectation, it may be better advised not
> to do so nevertheless. Particularly, the specification of uncertain (or
> erroneous) expectations in research evaluations may have detrimental
> effects (e.g., Rafols et al., 2012).
>
>
>
> We know this also from the discussion about using impact factors for the
> assessment of individual papers or institutional units across fields. One
> easily generates error without the possibility to specify the uncertainty
> because the error is not only in the measurement (methodological), but also
> in the conceptualization (theoretical).
>
>
>
> Best,
>
> Loet
>
>
>
> References:
>
> Guerrero-Bote, V. P., & Moya-Anegón, F. (2012). A further step forward in
> measuring journals’ scientific prestige: The SJR2 indicator. *Journal of
> Informetrics, 6*(4), 674-688.
>
> Leydesdorff, L., & Bornmann, L. (2011a). Integrated Impact Indicators (I3)
> compared with Impact Factors (IFs): An alternative design with policy
> implications. *Journal of the American Society for Information Science
> and Technology, 62*(11), 2133-2146. doi: 10.1002/asi.21609.
>
> Leydesdorff, L., & Bornmann, L. (2011b). How fractional counting affects
> the Impact Factor: Normalization in terms of differences in citation
> potentials among fields of science. *Journal of the American Society for
> Information Science and Technology, 62*(2), 217-229.
>
> Lundberg, J. (2007). Lifting the crown—citation z-score. *Journal of
> informetrics, 1*(2), 145-154.
>
> Mingers, J. (2014). Problems with SNIP. *Journal of Informetrics, 8*(4),
> 890-894.
>
> Moed, H. F. (2010). Measuring contextual citation impact of scientific
> journals. *Journal of Informetrics, 4*(3), 265-277.
>
> Opthof, T., & Leydesdorff, L. (2010). *Caveats* for the journal and field
> normalizations in the CWTS (“Leiden”) evaluations of research performance. *Journal
> of Informetrics, 4*(3), 423-430.
>
> Rafols, I., & Leydesdorff, L. (2009). Content-based and Algorithmic
> Classifications of Journals: Perspectives on the Dynamics of Scientific
> Communication and Indexer Effects. *Journal of the American Society for
> Information Science and Technology, 60*(9), 1823-1835.
>
> Rafols, I., Leydesdorff, L., O’Hare, A., Nightingale, P., & Stirling, A.
> (2012). How journal rankings can suppress interdisciplinary research: A
> comparison between innovation studies and business & management. *Research
> Policy, 41*(7), 1262-1282.
>
> Ruiz-Castillo, J., & Waltman, L. (2015). Field-normalized citation impact
> indicators using algorithmically constructed classification systems of
> science. *Journal of Informetrics, 9*(1), 102-117.
>
> Waltman, L., Van Eck, N. J., Van Leeuwen, T. N., Visser, M. S., & Van
> Raan, A. F. J. (2011). Towards a New Crown Indicator: Some Theoretical
> Considerations. *Journal of Informetrics, 5*(1), 37-47.
>
> Waltman, L., van Eck, N. J., van Leeuwen, T. N., & Visser, M. S. (2013). Some
> modifications to the SNIP journal impact indicator. *Journal of
> Informetrics, 7*(2), 272-285.
>
> Zitt, M., & Small, H. (2008). Modifying the journal impact factor by
> fractional citation weighting: The audience factor. *Journal of the
> American Society for Information Science and Technology, 59*(11),
> 1856-1860.
>
>
>
>
>  ------------------------------
>
> Loet Leydesdorff
>
> *Emeritus* University of Amsterdam
> Amsterdam School of Communications Research (ASCoR)
>
> loet at leydesdorff.net ; http://www.leydesdorff.net/
> Honorary Professor, SPRU, <http://www.sussex.ac.uk/spru/>University of
> Sussex;
>
> Guest Professor Zhejiang Univ. <http://www.zju.edu.cn/english/>,
> Hangzhou; Visiting Professor, ISTIC,
> <http://www.istic.ac.cn/Eng/brief_en.html>Beijing;
>
> Visiting Professor, Birkbeck <http://www.bbk.ac.uk/>, University of
> London;
>
> http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en
>
>
>
> *From:* Loet Leydesdorff [mailto:loet at leydesdorff.net
> <loet at leydesdorff.net>]
> *Sent:* Sunday, March 29, 2015 8:27 PM
> *To:* 'ASIS&T Special Interest Group on Metrics'
> *Subject:* RE: [SIGMETRICS] New Letter to the Editor
>
>
>
> In my opinion, the standard indicator in a field is defined by its
> frequency of professional use (and not by advantages and disadvantages of
> relevant indicators). In other words, if professional bibliometricians (and
> not amateur-bibliometricians) mostly use the MNCS (based on WoS subject
> categories), this is the standard then.
>
>
>
> Perhaps, this is an argument for “amateur-bibliometrics” J because the
> suggestion of normalization in professional bibliometrics is—as you
> claim—most of the time erroneous (e.g., Mingers, 2014).
>
>
>
> Best,
>
> Loet
>
>
>
>
>
> Reference:
>
> Mingers, J. (2014). Problems with SNIP. *Journal of Informetrics, 8*(4),
> 890-894.
>
>
>

-- 
Dr Jonathan Adams
Chief Scientist, Digital Science
Visiting Professor, King's College London
*http://www.kcl.ac.uk/sspp/policy-institute/people/kpi-visiting/adams.aspx
<http://www.kcl.ac.uk/sspp/policy-institute/people/kpi-visiting/adams.aspx>*

M/ +44 7964 908449
E/ j.adams at digital-science.com

Macmillan Publishers Ltd
The Glasshouse Building
2 Trematon Walk
(via Wharfdale Road)
London N1 9FN, UK

http://www.timeshighereducation.co.uk/news/research/research-intelligence-proof-is-in-the-numbers/411118.article

-- 
******************************************************************************** 

DISCLAIMER: This e-mail is confidential and should not be used by anyone 
who is
not the original intended recipient. If you have received this e-mail in 
error
please inform the sender and delete it from your mailbox or any other 
storage
mechanism. Neither Macmillan Digital Science Limited nor any of its agents 
accept
liability for any statements made which are clearly the sender's own and not
expressly made on behalf of Macmillan Digital Science Limited or one of its 
agents.
Please note that neither Macmillan Digital Science Limited nor any of its 
agents
accept any responsibility for viruses that may be contained in this e-mail 
or
its attachments and it is your responsibility to scan the e-mail and 
attachments (if any). No contracts may be concluded on behalf of Macmillan Digital 
Science Limited or its agents by means of e-mail communication. Macmillan 
Digital Science Limited Registered in England and Wales with registered 
number 7397265 
Registered Office Brunel Road, Houndmills, Basingstoke RG21 6XS   
********************************************************************************

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20150330/c5de5dd8/attachment.html>