[Eurchap] Using Web Citations to Measure Scientific Journal Impact

[Thelwall, Mike (Dr)]

Accepted abstract from the forthcoming ASIST-AoIR workshop
http://www.asis.org/Chapters/europe/announcements/AoIR.htm

Using Web Citations to Measure Scientific Journal Impact
Debora Shaw
School of Library and Information Science, Indiana University, 1320 E. 10th
Street, Main Library 011, Bloomington, IN 47405-3907, U.S.A.

Liwen Vaughan
Faculty of Information and Media Studies, University of Western Ontario,
London, Ontario, N6A 5B7, Canada

Entirely Web-based publication of scientific research, an alluring prospect,
may change entirely the nature of scientific communication, but is there
evidence that Web access can affect assessments of journal impact even in
our predominantly print-on-paper age? Could Web citation data supplement or
replace ISI-based measures of impact? Our studies show that Web citation
data demonstrate wider geographic influence of journal articles and that
these citations correlate with impact assessments using ISI data, especially
in disciplinary journals.

Building on to our earlier study of LIS journals, we extended the scope of
investigation to four science fields: biology, genetics, medicine, and
multidisciplinary sciences (for example, Nature, Science) to validate
earlier findings and to examine disciplinary difference. The inclusion of
wider varieties of journals from various countries allowed an exploration of
geographical and cultural impact manifested on different media (the Web vs.
traditional media) of scientific communication.

We conducted Google searches for Web citations (text citations of, not links
to) journal articles in the four fields. We recorded the number and nature
of the items retrieved, and then compared the number of Web citations with
the ISI citation counts. For a sample of 5,972 articles published in 114
journals, the median Web citation counts per journal article range from 6.2
in medicine to 10.4 in genetics; median ISI citation counts range from 1.4
for the multidisciplinary journals to 6.6 in genetics. It is clear that
actual numbers of Web citations are higher than ISI citation counts for all
four disciplines, suggesting that Web citations could provide more
fine-grained assessment of impact. In most cases, the number of ISI and Web
citations is correlated, indicating they are measuring the same thing. 

We narrowed the focus of the Google results, considering only Web citations
from research papers or class readings lists (these sources we took to
represent intellectual impact, as opposed to the perfunctory listing, for
example, of an article on a journal's home page or an author's c.v.). Even
when we look at only the Web citations which represent intellectual impact,
we find that ISI and Web citation counts are correlated. The percentage of
Web citations indicating intellectual impact is about 30% for each area.
Journals receiving more Web citations also have higher percentages of
citations indicating intellectual impact (e.g., 50% of Web citations to
Nature articles are of this type).

However, there are significantly lower correlations between Web and ISI
citations to articles in journals published outside the UK or USA. The Web
citations are generally more numerous than the ISI citation counts would
predict. This suggests that Web citations to non-UK/USA journals may be more
frequent in sources not covered by the ISI; i.e., the Web's wider linguistic
and geographic coverage reveals significantly more citations, coming from
different sources. Thus, Web citations may provide a more global assessment
of impact - the decentralised model to counter the centralisation of ISI
lamented, for example, in the Valparaiso Declaration for Improved Scientific
Communication in the Electronic Medium. 

The multidisciplinary science journals show lower correlation between Web
and ISI citation counts (45% of these journals show statistically
significant correlation, compared with over 60% for the other disciplines).
There is considerable variability in this category, with well-known
multidisciplinary journals (such as Nature and Science) showing higher
correlations (correlation coefficient 0.7 and 0.6 respectively), but lower
correlations for the numerous journals published outside the UK/USA or
available only in print.

As the dominance of print-based dissemination of scientific information is
challenged, the Web is increasing the number and diversity of voices
assessing contributions to the literature. It is reassuring that the
Web-derived assessments generally correlate with the judgments from
established evaluative techniques, notably ISI citation counts and journal
impact factors. This observation would seem to indicate that Web citation
counts could supplement or replace ISI measures of impact and could provide
an increased diversity in coverage. However, we caution that the
correlations between Web and ISI citations may be a transient phenomenon.
The ease of self-publishing and self-citing on the Web may allow knowing
authors to take advantage of the absence of review mechanisms to inflate
their citation counts. The ongoing debate about the nature and ethics of
self-citation can compound itself in many ways when the constraints of
editorial and peer review are removed.