Reply to Re: [SIGMETRICS] Impact Factor in Germany and other non-English countries

[Thed van Leeuwen]

Dear colleagues,

In the Sigmetrics mailing list we came across a reply by Eugene Garfield to
our letter in Research Evaluation vol.9 (2), p. 155-6 (Aug. 2000), titled
"First evidence of serious language-bias in the use of citation analysis
for the evaluation of national science systems", authored by Th.N. van
Leeuwen, H.F. Moed, R.J.W. Tijssen, M.S. Visser and A.F.J. van Raan. This
research letter is an extract of a full paper related to a presentation
given at the 6th International Conference on Science & Technology
Indicators in Leiden in May 2000. Evidently, we cannot present in the
letter all details of the presentation, nor the full paper. At the
conference we presented results from studies, indicating that the impact
scores of scientists from non-English speaking countries like Germany,
France and Switzerland changed drastically when we excluded the ISI-covered
publications written in other languages than English. This led to the
conclusion that language of publication is an important aspect in citation
analysis. Especially in the case of the comparison of national R&D systems,
as this phenomenon causes significant changes in the rankings of countries.
We did not find this phenomenon for just one Germany university or only one
faculty/school within a single university (as reported in the Research
Evaluation paper). The phenomenon described here was found for the entire
biomedical research systems of Germany, France, and Switzerland, and to a
lesser extent also for Italy, Spain and Japan.

We think there are a number of misunderstandings. The first relates to the
issue of the users of the ISI databases. Dr. Garfield states that these
databases are used as literature search tools (for example by scientists
and librarians), and therefore should not exclude non-English journal
publications. Of course we are aware of this 'original' function of the ISI
databases. Dr. Garfield argues that we propose to exclude these non-English
publications from the ISI databases. This is obviously not the case. We
merely emphasize that these databases are not only used as literature
search tools, but more and more serve as a source for the calculation of
research performance measures. And exactly here both worlds seem to
intersect, namely where scientists and librarians also use journal impact
measures, either for their publication strategy or their decision-making
with respect to their journal collection. And it is precisely this mixture
of applications that concerns us. The inclusion of certain journals and
their characteristics on one level of use - literature search - might
influence the use of the database for other purposes - performance
assessment -.

A second misunderstanding relates to Dr. Garfield's suggestion that what we
essentially want to promote 'our way' of doing citation analysis.
Unfortunately Dr. Garfield overlooks the point of our self-criticism here.
Until now, we have always analysed the publication data from non-English
Western countries such as Germany, France, without taking into account
differences in impact scores, caused by publication language. What we
suggest in our recent paper is to be aware of these problems, and most
certainly not to prescribe (if ever possible) to do it 'our way'. As
impartial scientists, we merely stress the need to develop more appropriate
research performance indicators. This causes new problems, like for
example, how to deal with low impact English journals. This problem was
explicitly stated at the abovementioned conference presentation, and
definitely will lead to further research on this topic.

A third issue relates to the interpretation by Dr. Garfield of the word
'bias'. What we try to achieve is to enlarge the awareness of people using
ISI data (both in 'raw' form, as well as their 'derivatives' such as
indicators) of the flaws related to this material. The fact that we use the
material ourselves is the best evidence of the absolute unique value of the
material. However, ISI database play a highly influential role in the world
of science, and we feel it our task to improve and, if necessarily,
criticize the material. In this particular case, we felt it our duty to
report these findings, because it creates a new perspective on the
comparison of national R&D systems, particularly of the countries in the
Western world. "Bias" is therefore used to create the necessary awareness
in the interpretation of impact scores based on publication sets that are
basically different of nature.


On behalf of the authors,

Thed van Leeuwen,
Centre for Science & Technology Studies (CWTS),
Leiden University, Leiden, the Netherlands

A pre-print of the paper related to the conference presentation is
presented at the CWTS website: http://cwts.leidenuniv.nl