ABS: Solari, A new approach to the SCI Journal Citation Reports, a system for evaluating scientific journals

[Gretchen Whitney]

Solari A, Magri MH, "A new approach to the SCI Journal Citation R eports,
a system for evaluating scientific journals "  Scientometrics, 47 (3)
605-625, March-April 2000

E-mail: magri at jouy.inra.fr

TITLE   :  A new approach to the SCI Journal Citation Reports, a system for
evaluating scientific journals
AUTHOR:  Solari A, Magri MH
JOURNAL: SCIENTOMETRICS,  47: (3) 605-625 MAR-APR 2000

Document type: Article    Language: English    Cited References: 16    Times
Cited: 0

Abstract:
The Science Citation Index, Journal Citation Reports (JCR), published by the
Institute for Scientific Information (ISI) and designed to rank, evaluate,
categorize and compare journals, is used in a wide scientific context as a
tool for evaluating researchers and research work, through the use of just
one of its indicators, the impact factor. With the aim of obtaining an
overall and synthetic perspective of impact factor values, we studied the
frequency distributions of this indicator using the box-plot method. Using
this method we divided the journals listed in the JCR into five groups (low,
lower central, upper central, high and extreme). These groups position the
journal in relation to its competitors. Thus, the group designated as
extreme contains the journals with high impact factors which are deemed to
be prestigious by the scientific community. We used the JCR data from 1996
to determine these groups, firstly for all subject categories combined (all
4779 journals) and then for each of the 183 ISI subject categories. We then
substituted the indicator value for each journal by the name of the group in
which it was classified. The journal group may differ from one subject
category to another. In this article, we present a guide for evaluating
journals constructed as described above. It provides a comprehensive and
synthetic view of two of the most used sections of the JCR, It makes it
possible to make more accurate and complete judgements on and through the
journals, and avoids an oversimplified view of the complex reality of the
world of journals. It immediately reveals the scientific subject category
where the journal is best positioned. Also, whereas it used to be difficult
to make intra- and interdisciplinary comparisons, this is now possible
without having to consult the different sections of the JCR. We construct
this guide each year using indicators published in the JCR by the ISI.

KeyWords Plus:
IMPACT FACTORS

Addresses:
Magri MH, INRA, Ctr Rech Jouy En Josas, Unite Cent Documentat, Stn Physiol
Anim, Equipe Stat, F-78352 Jouy En
Josas, France.
INRA, Ctr Rech Jouy En Josas, Unite Cent Documentat, Stn Physiol Anim,
Equipe Stat, F-78352 Jouy En Josas,
France.
INRA, Ctr Rech Jouy En Josas, Unite Cent Documentat, Unite Cent Document,
F-78352 Jouy En Josas, France.

Publisher:
KLUWER ACADEMIC PUBL, DORDRECHT

IDS Number:
304YG

ISSN:
0138-9130

CITED REFERENCES:
Cited Author            Cited Work                      Volume      Page
Year

NATURE NEUROSCIENCE                             1               641
1998
*I SCI INF            SCI CIT IND J CIT RE                      151
1997
 BASU A               SCIENTOMETRICS                44            347
1999
 CHAMBERS JM  GRAPHICAL METHODS DA                 395      1983
 ESCOFIER B            INITIATION TRAITEMEN                263      1997
 GARFIELD E            LIBRI                                    48
67      1998
 GILLIS JM             NAT NEUROSCI                       2       101
1999
 GLANZEL W             SCIENTOMETRICS             44        427      1999
 LEHMAN A              JMP STAT GRAPHICS GU                 19      1995
 MAGRI M               SCIENTOMETRICS                35          93
1996
 MAGRI MH              INFORMATION SCI TECH                71      1997
 MAGRI MH        VIE VALEUR VALORISAT                     225      1998
 MOED HF               J DOC                                         54
387     1998
 MOED HF               NATURE                           381       186
1996
 ROUSSEAU R            SCIENTOMETRICS              44        521     1999
 SCHWARTZ S            SCIENTOMETRICS              35        119     1996


__________________________________________________________________

        Within each subject category, the authors arrange journals into five
groups.  The "extreme" group (highest impact) consists of 371 journals (7.6%
of 4750 titles) with impact of 3.17 or higher.  The "high" group is 17.4%
with impact between 1.475 and 3.171.

        Reproduced below is the Conclusion of the paper
Conclusion

        "The Science Citation Index, Journal Citation Reports (JCR) is
regularly used in the process of evaluating research and researchers through
the journals' impact factor. The JCR has been published regularly every year
since 1974. This, together with its uniqueness, the way it is presented in
tables of numbers and the analysis provided by the classifications, make the
JCR an impartial, valid and essential reference. This role played by the JCR
in the evaluation process leads to confound the research work with the
journal in which it is published. Yet, assimilating the quality of a study
to the reputation of the journal in which it is published can be misleading
and have adverse effects which may only be seen much later. These effects
may be felt at different levels, such as funding withdrawal, failure to
succeed in competitive examinations or job applications, journal suppression
from the JCR then from the market. 9-11 The criticism often made concerning
the methodology inherent to the system, the process used for calculating the
impact factor combined with the results we have obtained with the JCR data,
lead us to believe not be used as such for evaluation purposes. 12,13 Other
journal evaluation techniques are proposed.14 17 These techniques  are also
based on citations, but are more complicated to apply than simply reading
values from a computer screen. The JCR is used in this way by many
decision-makers in research management, editorial policy and library
management. The simplicity of using  the impact factor may overshadow,
unfortunately, the progress being made in the field of bibliometric
research.

        Our guide for evaluating journals, which is the synthetic
reconstruction of two sections of the SCI JCR helps to avoid making
incorrect interpretations, because it avoids an over-simplified view of the
complex reality of the world of journals. The  and thereby the user can see
at a glance in which subject field the journal is best  positioned. Whereas
it used to be difficult to make comparisons between subject categories, it
is now possible to do so without the tedious and time-consuming consultation
of different JCR sections. Moreover, characterizing journals by the group in
which they are classified means that we are no longer conditioned by the
highest impact factor values, nor the lowest. By making their influence
relative, we are able to remove ourselves from the hegemony of journals
whose impact factors are outside values, and the evaluation practices based
on this type of indicator may thereby evolve. We often use this evaluation
guide, combined with other tools, to position research work, to choose
journals in which to publish, and to guide our journal acquisition policy
and editorial policy. A similar study has been carried out on other ISI
indicators.

        Nonetheless, our results concerning journals and subject categories
raise a number of questions. In a given subject category, what exactly is
the meaning of the presence of leading journals (those making up the
different extreme groups) and/or the presence of journals assigned to
several ISI subject categories? Does this presence reflect the dynamics of
the subject category, its multi-disciplinarity or specificity? The answer
partly lies in the problem of scientific subject category delimitation. As
Glänzel et al16 noted, "the delimitation of scientific subfields is one of
the central issues in both information science and bibliometric research.
The classification of scientific literature into appropriate subject fields
is, moreover, one of the basic preconditions of valid bibliometric
analyses". One possible way of finding answers to such different questions
would be to study the interdependence between ISI subject categories, and
then to determine the structure of the subject categories through the
journals listed within them.

        This study was carried out with the technical, efficient and
rigorous co-operation of M. Désiré. The article was translated by A-M.
Wall., translation and terminology Unit, INRA Jouy-en-Josas."

---------------------------------------

(c) ISI, Reprinted with permission
Please visit their website at www.isinet.com