New Letter to the Editor

Mon Mar 30 10:06:23 EDT 2015

Hi All-
This all makes sense to me – I follow the math and I understand the limitations but… In practical terms, I need to do an analysis that I can defend to senior engineers and that will likely get visibility at high levels.  In previous analyses, I have used % in WoS categories – top category only but the analysis was intended to get a feel for the publication venues. My next project is looking at the top (and I need to define that in a defensible way) publications from an institution probably since ~1980 but it would be even better to go back to ~1942. The institution publishes mostly in engineering (electrical, aerospace), but just enough things in biomedicine to make it obvious some normalization by field is needed. Level of publications is about 500/year. I have access to WoS and Scopus but just through the web interface – there’s no budget for leasing a larger data set.

There’s also the time since publication normalization to be considered. The immediacy and decay vary by field. In a similar study done in ~1986 for the same institution (hence the justification for going back to 1980), the authors considered a paper “frequently cited” if it was cited >150 times in 25 years, 120 citations in 15 years, or more than 50 citations in 5 years. The authors did not field normalize so the results were mostly from chemistry.

I am probably  - make that definitely – overthinking this but I want to deliver the highest quality work and I may get to publish this in the institution’s journal (which is in the 4th quartile in its category ☹ ).

Questions are:

1)      Given the weaknesses of the various methods, which do you recommend I use for field normalization?

2)      Which approach makes sense for time normalization? Maybe if article is older than cited half-life…

3)      What should I consider in combining these?  Does order matter? For processing time but substantively?

Thanks!

Christina

------
Christina K. Pikas
Librarian
The Johns Hopkins University Applied Physics Laboratory
Baltimore: 443.778.4812
D.C.: 240.228.4812
Christina.Pikas at jhuapl.edu

And still PhD candidate at Maryland…

From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Jonathan Adams
Sent: Monday, March 30, 2015 8:52 AM
To: SIGMETRICS at listserv.utk.edu
Subject: Re: [SIGMETRICS] New Letter to the Editor

Adminstrative info for SIGMETRICS (for example unsubscribe): http://web.utk.edu/~gwhitney/sigmetrics.html
I agree with Lutz's general point, that any reasonable methodology will usually produce similar results especially when applied to a reasonably large sample of reasonably balanced data. We all recognise that the underlying driver is that some teams/institutions/countries produce greater numbers of more frequently cited publications. You have to be perverse for them' not to 'do well'.

The problem that Loet is pointing us towards is that many analysts are applying methodology towards smaller samples, or less well-balanced data, and that they are teasing out factors regarding less 'peak' and more 'platform' performance. If they are delivering reports to a client or an employing organisation then the methodology (and interpretation) they use may have a significant and not always well-founded influence.

Jonathan Adams
Digital Science

On 30 March 2015 at 13:27, Bornmann, Lutz <lutz.bornmann at gv.mpg.de<mailto:lutz.bornmann at gv.mpg.de>> wrote:
Hi Loet,

I agree that we have very good alternatives for the MNCS and the use of WoS categories. However, the alternatives have their own (mostly practical) weaknesses. Furthermore, it seems that the different normalization methods produce similar results (see http://www.sciencedirect.com/science/article/pii/S1751157715000073).

Perhaps, other people on this list can report which normalization method they use (as standard). In my opinion, it would be interesting to know this.

Best,

Lutz

From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU<mailto:SIGMETRICS at LISTSERV.UTK.EDU>] On Behalf Of Loet Leydesdorff
Sent: Monday, March 30, 2015 9:17 AM
To: SIGMETRICS at LISTSERV.UTK.EDU<mailto:SIGMETRICS at LISTSERV.UTK.EDU>
Subject: Re: [SIGMETRICS] New Letter to the Editor

Adminstrative info for SIGMETRICS (for example unsubscribe): http://web.utk.edu/~gwhitney/sigmetrics.html
PS.

Both discussions – the one about using the mean (MNCS) and the one about using WoS Subject Categories for the normalization – seem now to have stagnated.

1.       Instead of the mean, one should use percentile rank classes. This was a step in a line of thought in 2010-2011 in which we first criticized the “old” crown indicator and then proposed what later became labeled by CWTS as MNCS (Opthof & Leydesdorff, 2010; cf. Lundberg, 2007; Waltman et al., 2011). We subsequently moved to percentiles, and automated the “Integrated Impact Indicator” that enables users to define one’s percentile rank classes at http://www.leydesdorff.net/software/i3 (Leydesdorff & Bornmann, 2011a).

Another line of thought was source-normalization or fractional counting of the citations (Zitt & Small, 2008; Moed, 2010; Leydesdorff & Bornmann, 2011b). This was elaborated into the SNIP and then into SNIP2. I mentioned Mingers (2014) because this development seems to have got stuck now; the critique does no longer matter?) SJR-2 (Guerrero-Bote et al., 2012), of course, provides an alternative, but nobody can use this indicator outside the institute that constructed it.

In my opinion, I3 and source-normalization (fractional counting) of the citations are still good ideas if one does not have WoS in-house through a license. Perhaps, this is an argument for what you call “amateur-bibliometrics”. It is better than taking the mean.

2.       In principle, SNIP and fractional counting creatively solve the determination of reference sets. The issue is not “normalization” per se, but the specification of an expectation (to be used in the denominator). The institutionalization in Scopus, however, may have been premature; or is there room to move to SNIP-3, and so forth? (Waltman et al., 2013). SNIP may be too technical to be reproduced (or controlled) outside the context of its production.

The determination of reference sets in terms of journals may not work or not be possible (Rafols & Leydesdorff, 2009). The sets are fuzzy and remain changing. CWTS now moved in the Leiden Rankings 2014 to direct clustering of the citations, but the 800+ fields can no longer be validated (Ruiz-Castillo & Waltman, 2015). A disadvantage is that nobody can reproduce the results outside the institute which constructed these “fields”. We know that algorithmic constructs do not necessarily match with intellectual classifications. Furthermore, because the delineation is paper-based (instead of journal-based), one would have to update continuously. Thus, the “fields” cannot be reproduced at a next moment of time.

If one is not able to specify an expectation, it may be better advised not to do so nevertheless. Particularly, the specification of uncertain (or erroneous) expectations in research evaluations may have detrimental effects (e.g., Rafols et al., 2012).

We know this also from the discussion about using impact factors for the assessment of individual papers or institutional units across fields. One easily generates error without the possibility to specify the uncertainty because the error is not only in the measurement (methodological), but also in the conceptualization (theoretical).

Best,
Loet

References:
Guerrero-Bote, V. P., & Moya-Anegón, F. (2012). A further step forward in measuring journals’ scientific prestige: The SJR2 indicator. Journal of Informetrics, 6(4), 674-688.
Leydesdorff, L., & Bornmann, L. (2011a). Integrated Impact Indicators (I3) compared with Impact Factors (IFs): An alternative design with policy implications. Journal of the American Society for Information Science and Technology, 62(11), 2133-2146. doi: 10.1002/asi.21609.
Leydesdorff, L., & Bornmann, L. (2011b). How fractional counting affects the Impact Factor: Normalization in terms of differences in citation potentials among fields of science. Journal of the American Society for Information Science and Technology, 62(2), 217-229.
Lundberg, J. (2007). Lifting the crown—citation z-score. Journal of informetrics, 1(2), 145-154.
Mingers, J. (2014). Problems with SNIP. Journal of Informetrics, 8(4), 890-894.
Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265-277.
Opthof, T., & Leydesdorff, L. (2010). Caveats for the journal and field normalizations in the CWTS (“Leiden”) evaluations of research performance. Journal of Informetrics, 4(3), 423-430.
Rafols, I., & Leydesdorff, L. (2009). Content-based and Algorithmic Classifications of Journals: Perspectives on the Dynamics of Scientific Communication and Indexer Effects. Journal of the American Society for Information Science and Technology, 60(9), 1823-1835.
Rafols, I., Leydesdorff, L., O’Hare, A., Nightingale, P., & Stirling, A. (2012). How journal rankings can suppress interdisciplinary research: A comparison between innovation studies and business & management. Research Policy, 41(7), 1262-1282.
Ruiz-Castillo, J., & Waltman, L. (2015). Field-normalized citation impact indicators using algorithmically constructed classification systems of science. Journal of Informetrics, 9(1), 102-117.
Waltman, L., Van Eck, N. J., Van Leeuwen, T. N., Visser, M. S., & Van Raan, A. F. J. (2011). Towards a New Crown Indicator: Some Theoretical Considerations. Journal of Informetrics, 5(1), 37-47.
Waltman, L., van Eck, N. J., van Leeuwen, T. N., & Visser, M. S. (2013). Some modifications to the SNIP journal impact indicator. Journal of Informetrics, 7(2), 272-285.
Zitt, M., & Small, H. (2008). Modifying the journal impact factor by fractional citation weighting: The audience factor. Journal of the American Society for Information Science and Technology, 59(11), 1856-1860.

________________________________
Loet Leydesdorff
Emeritus University of Amsterdam
Amsterdam School of Communications Research (ASCoR)
loet at leydesdorff.net <mailto:loet at leydesdorff.net> ; http://www.leydesdorff.net/
Honorary Professor, SPRU, <http://www.sussex.ac.uk/spru/> University of Sussex;
Guest Professor Zhejiang Univ.<http://www.zju.edu.cn/english/>, Hangzhou; Visiting Professor, ISTIC, <http://www.istic.ac.cn/Eng/brief_en.html> Beijing;
Visiting Professor, Birkbeck<http://www.bbk.ac.uk/>, University of London;
http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en

From: Loet Leydesdorff [mailto:loet at leydesdorff.net]
Sent: Sunday, March 29, 2015 8:27 PM
To: 'ASIS&T Special Interest Group on Metrics'
Subject: RE: [SIGMETRICS] New Letter to the Editor

In my opinion, the standard indicator in a field is defined by its frequency of professional use (and not by advantages and disadvantages of relevant indicators). In other words, if professional bibliometricians (and not amateur-bibliometricians) mostly use the MNCS (based on WoS subject categories), this is the standard then.

Perhaps, this is an argument for “amateur-bibliometrics” ☺ because the suggestion of normalization in professional bibliometrics is—as you claim—most of the time erroneous (e.g., Mingers, 2014).

Best,
Loet

Reference:
Mingers, J. (2014). Problems with SNIP. Journal of Informetrics, 8(4), 890-894.

--
Dr Jonathan Adams
Chief Scientist, Digital Science
Visiting Professor, King's College London
http://www.kcl.ac.uk/sspp/policy-institute/people/kpi-visiting/adams.aspx

M/ +44 7964 908449
E/ j.adams at digital-science.com<mailto:j.adams at digital-science.com>

Macmillan Publishers Ltd
The Glasshouse Building
2 Trematon Walk
(via Wharfdale Road)
London N1 9FN, UK

http://www.timeshighereducation.co.uk/news/research/research-intelligence-proof-is-in-the-numbers/411118.article

********************************************************************************
DISCLAIMER: This e-mail is confidential and should not be used by anyone who is
not the original intended recipient. If you have received this e-mail in error
please inform the sender and delete it from your mailbox or any other storage
mechanism. Neither Macmillan Digital Science Limited nor any of its agents accept
liability for any statements made which are clearly the sender's own and not
expressly made on behalf of Macmillan Digital Science Limited or one of its agents.
Please note that neither Macmillan Digital Science Limited nor any of its agents
accept any responsibility for viruses that may be contained in this e-mail or
its attachments and it is your responsibility to scan the e-mail and
attachments (if any). No contracts may be concluded on behalf of Macmillan Digital Science Limited or its agents by means of e-mail communication. Macmillan
Digital Science Limited Registered in England and Wales with registered number 7397265
Registered Office Brunel Road, Houndmills, Basingstoke RG21 6XS
********************************************************************************

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20150330/c81294cf/attachment.html>