How to compare research impact of toll- vs. open -access research

[Stevan Harnad]

On Wed, 14 Apr 2004, Garfield, Eugene wrote:

> The results obtained for computer science by analysis of CiteSeer are
> distorted for a variety of reasons. They cannot be compared with the
> literature of e.g. life sciences. Computer science is heavily dependent upon
> conference literature. I cannot comment upon the physics literature, but
> there are other studies which seem to indicate that readership increases
> will not necessarily be followed by increased citation impact.

Gene Garfield, the father of citation analysis as well as ISI, is quite
right that the Lawrence (2001) study on the impact enhancing effects of
open access in computer science needed to be replicated in other fields
to check whether it was merely an artifact of the fact that computer
science is conference- rather than journal-based.

    Lawrence, S. (2001) Online or Invisible? Nature 411 (6837): 521.
    http://www.neci.nec.com/~lawrence/papers/online-nature01/

But, thanks to the ISI database licensed to OST and a special contract
generously provided by ISI to conduct the study, we are in the process of
testing the Lawrence effect across all disciplines in a 10-year ISI sample
of 14 million articles. The physics analyses up to 2001 are already done,
and they reveal even larger effects than those reported by Lawrence,
with OA/non-OA citation ratios of 2.5 - 5.8. All indications are that 2002
will raise them even further, as the biggest effects occur within the
first 3 years of publication in scientific disciplines (and both OA and
the awareness and visibility of OA articles are also increasing yearly).

    Brody, T., Stamerjohanns, H., Harnad, S. Gingras, Y. & Oppenheim,
    C. (2004) The effect of Open Access on Citation Impact. Presented at:
    National Policies on Open Access (OA) Provision for University
    Research Output: an International meeting, Southampton,
    19 February 2004. http://opcit.eprints.org/feb19prog.html
    http://www.ecs.soton.ac.uk/~harnad/Temp/OATAnew.pdf

> In one study of a single chemical journal that I refereed there were about
> 100 readerships for each citation of that journal, but there did not seem to
> be any perceptible increase of citation by the research literature.

The ratio of "reads" to "cites" will no doubt vary by field. Kurtz and
co-workers report it as 17:1 and even 12:1 in astrophysics.

    Kurtz, Michael J.; Eichhorn, Guenther; Accomazzi, Alberto; Grant,
    Carolyn S.; Demleitner, Markus; Murray, Stephen S.; Martimbeau,
    Nathalie; Elwell, Barbara.  (2003) The NASA Astrophysics Data
    System: Sociology, Bibliometrics, and Impact.  Journal of
    the American Society for Information Science and Technology
    http://cfa-www.harvard.edu/~kurtz/jasis-abstract.html

    Kurtz, M.J. (2004) Restrictive access policies cut readership of
    electronic research journal articles by a factor of two, Michael
    J. Kurtz, Harvard-Smithsonian Centre for Astrophysics, Cambridge,
    MA http://opcit.eprints.org/feb19oa/kurtz.pdf

Tim Brody's remarkable download/citation correlator/predictor gives the
size of the correlation by field, and can be used to predict citation
6-24 months later from downloads today (with an adjustable time-window):

    http://citebase.eprints.org/analysis/correlation.php

> Undoubtedly the web will increase apparent readership of literature, but
> that will not necessarily change the population of relevant researchers who
> are in a position to cite particular studies.

Not necessarily, but very probably! And also actually, in the fields
tested so far. After all, access is a necessary if not a sufficient
precondition for citation. And since Open Access (OA) dramatically increases
the number of would-be users who would otherwise have been denied access
to the article (maximises it, in fact, for all who have access to the web)
it stands to reason that it can only increase both usage and impact.

The way to test this, however, is not just to compare apples and oranges (i.e.,
OA and non-OA journals). The right way is to compare OA and non-OA articles in
the *same* journals (and years). That is what our study with the ISI data is
doing.

(As Gene himself has often stressed, it is the article [and author]
citation counts that should be weighed, and not just the average
citation counts of the journals in which the article appears!)

> I do not think the ISI study is definitive but it is not irrelevant.

It is certainly not irrelevant to have shown "that there was no
discernible difference in terms of citation impact or frequency" between
the 191 OA journals and the 8509 non-OA journals indexed by ISI, equating
for comparable journals as closely as possible: http://www.isinet.com/oaj
But obviously there is a certain risk of circularity in this! It does
show that the skeptics are wrong (for these OA journals): OA journals
*are* indexed by ISI, and they *do* have comparable citation impacts.

But the real test of the effect of OA on citation (and download) impact
is on an article basis, where the journals can be equated exactly (by
being the *same* journal and year!). And there the impact-maximising
effects of OA are proving to be very dramatic indeed.

Stevan Harnad

> > Stevan Harnad wrote:
> >
> > The ISI press release says:
> >
> >     "Today, Thomson ISI... announced that journals published in the
> >      new Open Access (OA) model are beginning to register impact in
> >      the world of scholarly research...  Of the 8,700 selected journals
> >      currently covered in Web of Science, 191 are OA journals... [A
> >      study on] whether OA journals perform differently from other
> >      journals in their respective fields [found] that there was no
> >      discernible difference in terms of citation impact or frequency
> >      with which the journal is cited."  http://www.isinet.com/oaj
> >
> >   But if you want to get a better idea of the effect of OA on impact,
> >   don't just compare the 2% of ISI journals that are OA journals
> >   with the 98% that are not, to find that they are equal in impact
> >   (for this may well be comparing apples with oranges). Compare the
> >   much higher percentage of *articles* from the 98% non-OA journals
> >   that have been made OA by their authors -- by self-archiving
> >   them -- with articles (from the very same journals and volumes)
> >   that have *not* been made OA by their authors: You will find that
> >   there is indeed a discernible difference in terms of frequency
> >   with which the *article* is cited, and that that difference
> >   is from 250%-550% in favor of the articles that their authors
> >   have made OA! That is what an ongoing series of comparisons
> >   based on a 10-year sample of the same ISI database across all
> >   disciplines is revealing (in computer science and physics so far):
> >   http://www.ecs.soton.ac.uk/~harnad/Temp/OA-TAadvantage.pdf