Measuring cumulating research impact loss across fields and time

Wed Nov 26 07:09:47 EST 2003

On Wed, 26 Nov 2003, David Spurrett & Subbiah Arunachalam wrote:

>ds> I look forward to the results of the empirical study you describe.
>ds> I would be curious to know... whether
>ds> there was a further pattern that related (a) the extent to which
>ds> publications by authors at particular institutions cited research
>ds> materials available through open access, with (b) their local
>ds> institutional budget for expenditure on journals.
>
>sa> Stevan Harnad talked about a study on the relative
>sa> citation rates of open-access and toll-access articles
>sa> he is conducting in collaboration with UQaM,
>sa> Southampton, Oldenburg and Loughborough. When will the
>sa> results become available? Will there be any interim
>sa> reports? I am curious to know.

The study is ongoing and we will report the results (as a pre-refereeing
preprint!) as soon as they are available. But meanwhile, much information
inheres in -- and many telling estimates can be made from -- the data
that are already available.

David Spurrett's & Subbiah Arunachalam's queries suggest the following
preliminary analysis, which can already be done by anyone on the basis
of the data already available. (I will ask our super-talented team at
Southampton if they can squeeze it in, along with all the other ongoing
studies!):

We know from the Lawrence study (below) that the citation enhancement factor
for open- vs. toll-access is about 4.5 in computer science (4.5 times as many
citations for open- vs. toll-access articles in the same venue).
http://www.ecs.soton.ac.uk/~harnad/Temp/self-archiving_files/Slide0006.gif

We know from the Eysenck and Smith RAE outcome study in Psychology
(and from the Oppenheim studies in other disciplines) that the correlation
between RAE outcome and citation impact is about .90 (in Psychology).
http://www.ecs.soton.ac.uk/~harnad/Temp/self-archiving_files/Slide0007.gif

We also know the 2001 RAE outcome, rank-ordering every department in every
university in the UK http://www.hero.ac.uk/rae/submissions/ and we also
know the size of the funding and the funding difference associate with
each rank.

Hence it is very easy to take those rank orders, for each discipline,
and calculate -- based on that discipline's correlation between its
RAE rank and its citation impact -- the estimated income increase that
would arise from the rank increase induced by the impact increase
caused by open access!

In particular, it would be possible to illustrate how the rank order
would change if, for example, the research output of the lowest-ranked
department in each discipline became open-access, and gained a  2-fold,
3-fold, 4-fold, or 4.5-fold increase in impact (depending on how
close it came to the Lawrence 4.5 estimate -- which might itself be
an underestimate in some disciplines!). The RAE/impact correlation
would predict what rank that department would get, and the RAE/funding
correlation would predict how much more money that would translate into.

Obviously if *all* the articles in all disciplines suddenly became
open-access overnight, there would not be such a dramatic change in
rankings (though it would give some research a better fighting chance),
because all impact would simply be scaled up. (*Simply scaled up*! But
that in itself would represent a huge benefit to research progress and
productivity.)

But never mind that. We must appeal to our lower instincts, in trying
to persuade individual researchers and their institutions that open
access is in their interests. So the above data should be taken in
a first-come, first-served competitive spirit: Right now, it is
definitely not the case that *all* articles are open access. Almost all
are not. Nor is the transition happening overnight (as it could have done,
already a decade ago).
http://www.ecs.soton.ac.uk/~harnad/Temp/self-archiving_files/Slide0024.gif

So the incentive to self-archive comes from the fact that those
who do it *now* stand the best chance of changing the relative
research impact-ranking (and hence the research funding) in their
favor: and the study I've sketched would estimate by just how much.
A dimensionless picture of the size of the increment is already
visible in:
http://www.ecs.soton.ac.uk/~harnad/Temp/self-archiving_files/Slide0025.gif

The RAE data are open-access, so anyone can do this study. But I will try
to persuade the Southampton team to do it, in order to provide ammunition
for those who are hard at working trying to inform university
administrators and research funders about the benefits to be expected
from mandating open-access provision for all their research output.

[A slight correction to David Spurrett's query about the correlation between
>   "(a) the extent to which publications by authors at particular
>   institutions cited research materials available through open access,
>   with (b) their local institutional budget for expenditure on journals."
First, that's the wrong correlation. We've agreed it's not journal
budget expenditures that will persuade researchers to self-archive,
but research income. Second, we can already answer the question: That
correlation is zero, because the small existing volume of open-access
there is so far has not led to any toll-cancellations, in any discipline
(including Physics, where self-archiving and open-access are most advanced).
The correlation *might* change eventually, but that will not be a *cause* of
universal open access, but an *effect*:
http://www.nature.com/nature/debates/e-access/Articles/harnad.html#B1 ]

    Lawrence, S. (2001) Free online availability substantially
    increases a paper's impact. Nature Web Debates.
    http://www.nature.com/nature/debates/e-access/Articles/lawrence.html

    Kurtz, Michael J.; Eichhorn, Guenther; Accomazzi, Alberto;
    Grant, Carolyn S.; Demleitner, Markus; Murray, Stephen S.;
    Martimbeau, Nathalie; Elwell, Barbara. (submitted) The NASA
    Astrophysics Data System: Sociology, Bibliometrics, and Impact.
    http://cfa-www.harvard.edu/~kurtz/jasis-abstract.html

    the forthcoming Schwartz et al. study
    http://listserv.nd.edu/cgi-bin/wa?A2=ind0311&L=pamnet&D=1&O=D&P=1632

    the work of Andrew Odlyzko:
    http://www.dtc.umn.edu/~odlyzko/doc/complete.html

    and Tim Brody's remarkable citebase usage and citation impact calculator
    http://citebase.eprints.org/cgi-bin/search as well as his usage/citation
    impact correlator
    http://citebase.eprints.org/analysis/correlation.php
    which can predict later citation impact from earlier usage (download)
    impact using variable time-windows and ranges for the Physics ArXiv
    (you need the latest java to be able to use it) at:

    Smith, Andrew, & Eysenck, Michael (2002) "The correlation
    between RAE ratings and citation counts in psychology," June 2002
    http://psyserver.pc.rhbnc.ac.uk/citations.pdf

    Oppenheim, Charles (1995) The correlation between citation counts and
    the 1992 Research Assessment Exercises ratings for British library and
    information science departments, Journal of Documentation, 51:18-27.

    Oppenheim, Charles (1998) The correlation between
    citation counts and the 1992 research assessment exercise
    ratings for British research in genetics, anatomy
    and archaeology, Journal of Documentation, 53:477-87.
    http://dois.mimas.ac.uk/DoIS/data/Articles/julkokltny:1998:v:54:i:5:p:477-487.html

    Holmes, Alison & Oppenheim, Charles (2001) Use of citation analysis
    to predict the outcome of the 2001 Research Assessment Exercise for
    Unit of Assessment (UoA) 61: Library and Information Management.
    http://www.shef.ac.uk/~is/publications/infres/paper103.html

Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated online
RAE CVs Linked to University Eprint Archives: Improving the UK Research
Assessment Exercise whilst making it cheaper and easier.
Ariadne. http://www.ariadne.ac.uk/issue35/harnad/

Stevan Harnad

NOTE: A complete archive of the ongoing discussion of providing open
access to the peer-reviewed research literature online is available at
the American Scientist September Forum (98 & 99 & 00 & 01 & 02 & 03):
    http://amsci-forum.amsci.org/archives/september98-forum.html
    http://www.cogsci.soton.ac.uk/~harnad/Hypermail/Amsci/index.html
    Post discussion to: september98-forum at amsci-forum.amsci.org

Dual Open-Access Strategy:
    BOAI-2 ("gold"): Publish your article in a suitable open-access
            journal whenever one exists.
    BOAI-1 ("green"): Otherwise, publish your article in a suitable
            toll-access journal and also self-archive it.
    http://www.soros.org/openaccess/read.shtml
    http://www.ecs.soton.ac.uk/~harnad/Temp/berlin.htm
http://www.ecs.soton.ac.uk/~harnad/Temp/self-archiving_files/Slide0026.gif
http://www.ecs.soton.ac.uk/~harnad/Temp/self-archiving_files/Slide0021.gif
http://www.ecs.soton.ac.uk/~harnad/Temp/self-archiving_files/Slide0024.gif
http://www.ecs.soton.ac.uk/~harnad/Temp/self-archiving_files/Slide0028.gif