CWTS Journal Indicators

Thu Sep 26 05:48:05 EDT 2013

Dear Nees Jan,
I like the logic of the stability & reliability of the indicators, like CWTS applies here and in the 2013 Leidenranking (http://www.leidenranking.com/methodology/indicators).
I was wondering if we can connect some of the logic on reliability: for example:
Do journals in the highest decile (D1) of WOS subject categories also have a high reliability for the SNIP value per publication?
In other words: Can we use evidence from the WOS logic and combine it with the Scopus logic in some way or is this a stupid idea to mix journal level logic with article level logic?
It would be relevant if we could simply explain the combined force of the two logics to academic leaders/decisionmakers in universities who have to deal with the judgement on quality and excellence.
Wilfred Mijnhardt
@wmijnhardt

On 26/09/2013 11:03, Nees Jan van Eck wrote:
Dear Eric,

Thank you for your feedback on CWTS Journal Indicators.

We haven't documented the bootstrapping technique applied to the SNIP indicator in a paper. This is because it is a standard statistical technique. There is nothing new in our application of this technique. As you probably know, we also use the bootstrapping technique in the construction of stability intervals in the CWTS Leiden Ranking. In our paper on this ranking (http://dx.doi.org/10.1002/asi.22708), we briefly explain the technique (p. 2429) and we offer some literature references.

We appreciate your idea of adding a column with the field(s) to which a journal belongs. We will update this in the next few weeks.

Your remark on the Journal of Engineering Education is a valuable one. It allows me to explain some of the difficulties of calculating journal impact indicators based on a large multidisciplinary database (Scopus) that includes not only major international scientific journals but also a large number of other types of sources, such as trade journals, conference proceedings, and more locally oriented scientific journals. Part of the SNIP philosophy (but the same applies to the CWTS Leiden Ranking as well) is that in the calculation of field-normalized impact indicators one's starting point should not simply be the entire bibliographic database one has available (Scopus or Web of Science) but rather a selection of sources within one's database that, based on certain criteria, can be considered to be reasonably comparable to each other. For instance,  an international scientific journal and a trade journal targeted at an industrial audience can hardly be considered comparable to each other. What is needed is a consistent set of sources selected based on clear criteria (rather than based on the not so clear criteria used by database producers). In the case of SNIP, we have chosen to work with all sources targeted at a scientific audience (i.e., no trade journals) that give at least a certain minimum amount of references to other sources. In the SNIP calculation, sources that do not satisfy these criteria are excluded. Citations originating from these sources are not counted (but a SNIP value can still be calculated for these sources).

Why does the Journal of Engineering Education display such a large SNIP increase between 2007 and 2008? It turns out that the Journal of Engineering Education receives a considerable share of its citations from the proceedings of the ASEE Annual Conference and Exposition. Publications in these proceedings tend to have only a small number of references (within the three-year citation window used by SNIP), but the number of references has increased over time. For this reason, before 2008, the proceedings of the ASEE Annual Conference and Exposition were excluded from the SNIP calculation, but starting from 2008 they met our inclusion criteria (at least 20% of the publications have at least one 'active' reference) and were included in the calculation. This resulted in a large increase in the SNIP value of the Journal of Engineering Education. As a consequence of the inclusion of the proceedings of the ASEE Annual Conference and Exposition in 2008, the Journal of Engineering Education not only received more citations, but because the additional citations originate from a source with relatively few references, they have a high weight in the source normalization approach taken by SNIP.

I hope the above explanation clarifies the case of the Journal of Engineering Education. Of course, we may have further discussion on the appropriateness of the criteria used for selecting the sources to be included in the SNIP calculation. We welcome any suggestions on alternative criteria that could be considered.

Best regards,
Nees Jan

From: Éric Archambault [mailto:eric.archambault at science-metrix.com]
Sent: Wednesday, September 25, 2013 6:19 PM
To: Eck, N.J.P. van; SIGMETRICS at LISTSERV.UTK.EDU<mailto:SIGMETRICS at LISTSERV.UTK.EDU>
Subject: RE: [SIGMETRICS] CWTS Journal Indicators

Dear Nees Jan

Thank you for this addition to the growing list of journal indicators. Having a publicly accessible list of scores like this is really important and will play an important role in the debate on journal impact. Having rigorous researchers such as the ones at CWTS pursuing this project initiated by Michel Zitt and Henry Small and pursued by Moed is certainly useful.

However, I feel this is still at the stage of a research project and we should be careful to characterize our indicators carefully before telling the wider community that they are ready for prime time. We can't afford to have any more flaky journal impact indicators. This is now the forth proposition for such an indicator, after the Journal Impact Factor, the Scimago indicator, and the Eigenfactor. In this context, allowing both practitioners and users to decide which one seems to have the greatest scientific merit is essential. This requires that the methods and all ingredients be known to users and practitioners. Your paper is useful to understand the recipe but some ingredients are missing from the public disclosure and these need to be made public to help the community characterize your tool.

In particular, I think a few more details on the methods would be useful here. Firstly, having more details about the bootsrapping method that you use to compute the stability intervals would be welcome. Have you written a paper on this technique? Secondly, an additional column with the field of each journal would be more transparent and useful to users.

Do you have an explanation for this behavior:
The Journal of Engineering Education is one of the source journals with the highest SNIP in 2008. Is this an artifact or in 2008 the journal became that good compared to 2007? I find the jump surprising as it is outside the boundaries that you calculated. Of course, it is not impossible by chance to fall outside these, just that the jump is somewhat large.

Kind regards

Eric

Source title

Source type

Print ISSN

Year

P

SNIP

SNIP (lower bound)

SNIP (upper bound)

% self cit

Journal of Engineering Education

Journal

1069-4730

2002

207

7.901979

6.625

9.355

16%

Journal of Engineering Education

Journal

1069-4730

2003

215

6.587213

5.393

7.838

11%

Journal of Engineering Education

Journal

1069-4730

2004

194

9.710727

7.719

11.838

9%

Journal of Engineering Education

Journal

1069-4730

2005

133

2.498504

1.685

3.463

48%

Journal of Engineering Education

Journal

1069-4730

2006

111

4.458215

3.042

6.121

16%

Journal of Engineering Education

Journal

1069-4730

2007

98

6.650165

4.437

9.274

23%

Journal of Engineering Education

Journal

1069-4730

2008

93

20.62702

14.286

28.396

10%

Journal of Engineering Education

Journal

1069-4730

2009

75

15.92148

12.191

20.305

16%

Journal of Engineering Education

Journal

1069-4730

2010

77

16.12523

12.181

20.454

14%

Journal of Engineering Education

Journal

1069-4730

2011

76

16.1012

11.783

21.15

14%

Journal of Engineering Education

Journal

1069-4730

2012

90

12.49939

9.933

15.098

7%

From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Nees Jan van Eck
Sent: September-25-13 10:09 AM
To: SIGMETRICS at LISTSERV.UTK.EDU<mailto:SIGMETRICS at LISTSERV.UTK.EDU>
Subject: [SIGMETRICS] CWTS Journal Indicators

ml>
The 2012 SNIP values have been released on CWTS Journal Indicators (www.journalindicators.com<http://www.journalindicators.com>). SNIP (source normalized impact per paper) is a freely available journal impact indicator that uses a source normalization mechanism to correct for differences in citation practices between fields of science. Compared with the journal impact factor, SNIP allows for more accurate comparisons between journals active in different scientific fields. SNIP is calculated by CWTS based on Elsevier's Scopus database. With the release of the 2012 SNIP values, stability intervals have been added to CWTS Journal Indicators. These intervals indicate the reliability of the SNIP value of a journal. For instance, if a journal's SNIP value is largely due to a single very highly cited publication, this is indicated by a wide stability interval. SNIP is the only freely available journal impact indicator that is presented with stability intervals.

Your feedback on CWTS Journal Indicators is greatly appreciated.

Best regards,
Nees Jan van Eck

========================================================
Nees Jan van Eck PhD
Researcher
Head of ICT

Centre for Science and Technology Studies
Leiden University
P.O. Box 905
2300 AX Leiden
The Netherlands

Willem Einthoven Building, Room B5-35
Tel:       +31 (0)71 527 6445
Fax:       +31 (0)71 527 3911
E-mail:    ecknjpvan at cwts.leidenuniv.nl<mailto:ecknjpvan at cwts.leidenuniv.nl>
Homepage:  www.neesjanvaneck.nl<http://www.neesjanvaneck.nl/>
VOSviewer: www.vosviewer.com<http://www.vosviewer.com/>
========================================================

Disclaimer
________________________________
De informatie verzonden in dit e-mail bericht inclusief de bijlage(n) is vertrouwelijk en is uitsluitend bestemd voor de geadresseerde van dit bericht. Lees verder: www.eur.nl/email-disclaimer <http://www.eur.nl/email-disclaimer>
The information in this e-mail message is confidential and may be legally privileged. Read more: www.eur.nl/english/email-disclaimer<http://www.eur.nl/english/email-disclaimer>
________________________________

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20130926/f356c1e0/attachment.html>