Craig et al.'s review of the OA citation advantage

[Stevan Harnad]

                    ** Cross-Posted **

    Craig, Ian; Andrew Plume, Marie McVeigh, James Pringle & Mayur
    Amin (2007) Do Open Access Articles Have Greater Citation Impact?
    A critical review of the literature. Journal of Informetrics.
    http://www.publishingresearch.net/Citations-SummaryPaper3_000.pdf.pdf

I've read Craig et al.'s critical review ("proposed by the Publishing
Research Consortium") concerning the OA citation Impact effect and
will shortly write a short, mild review. But first here is a commentary
from Bruce Royan, followed by Sally Morris's posting, followed by a few
remarks from me. --  Stevan Harnad

>    ---------- Forwarded message ----------
>    Date: Sun, 20 May 2007 08:00:02 +0100
>    From: Bruce Royan <bruce.royan-- concurrentcomputing.co.uk>
>    To: diglib--infoserv.inist.fr
>    Subject: RE: [DIGLIB] Recent research tempers citation advantage of open
>           access
>    
>    Hmm. 
>    
>    Sally claims that according to this article "the relationship between
>    open access and citation, once thought to be almost self-evident,
>    has almost disappeared."

>    Now I'm no Informetrician, but my reading of the article is that
>    the authors reluctantly acknowledge that Open Access articles do
>    have greater citation impact, but claim that this is less because
>    they are Open Access per se, and more because:

>               -they are available sooner than more conventionally
>               published articles, or

>               -they tend to be better articles, by more prestigious
>               authors

>    Sally's point of view is understandable, since she is employed by a
>    consortium of conventional publishers. It's interesting to note that
>    the employers of the authors of this article are Wiley-Blackwell,
>    Thomson Scientific, and Elsevier.

>    Even more interesting is that, though this article has been accepted
>    for publication in the conventional "Journal of Informetrics", a pdf
>    of it (described as a summary, but there are 20 pages in JOI format,
>    complete with diagrams, references etc) has already been mounted on
>    the web for free download, in what might be mistaken for an example
>    of green route open access.

>    Could this possibly be in order to improve the article's impact?
>    
>    Professor Bruce Royan   http://www.linkedin.com/in/bruceroyan
>    Concurrent Computing Limited.     Registered Office:
>    Wellington House, Aylesbury Rd, Princes Risborough, Bucks HP27 0JP  
>
>From: Sally Morris info--publishingresearch.net
>Sent: 17 May 2007 18:00
>Subject: [DIGLIB] Recent research tempers citation advantage of open access
>    
>    'Do Open Access Articles Have Greater Citation Impact?  
>    A critical review of the literature'
>    Ian Craig, Andrew Plume, Marie McVeigh, James Pringle and Mayur Amin.
>    
>A new, comprehensive review of recent bibliometric literature finds
>decreasing evidence for an effect of 'Open Access' on article citation
>rates. The review, now accepted for publication in the Journal of
>Informetrics, was proposed by the Publishing Research Consortium (PRC) and
>is available at its web site at www.publishingresearch.net. It traces the
>development of this issue from Steve Lawrence's original study in Nature in
>2001 to the most recent work of Henk Moed and others.
>    
>Researchers have delved more deeply into such factors as 'selection bias'
>and 'early view' effects, and began to control more carefully for the
>effects of disciplinary differences and publication dates. As they have
>applied these more sophisticated techniques, the relationship between open
>access and citation, once thought to be almost self-evident, has almost
>disappeared.
>    
>Commenting on the paper, Lord May of Oxford, FRS, past president of the
>Royal Society, said 'In December 2005, the Royal Society called for an
>evidence-based approach to the scholarly communications debate. This
>excellent paper demonstrates that there is actually little evidence of a
>citation advantage for open access articles.'
>    
>The debate will certainly continue, and further studies will continue to
>refine current work. The PRC welcomes this discussion, and hopes that this
>latest paper may be a catalyst for a new round of informed scholarly
>exchange.  
>    
>Sally Morris
>on behalf of the Publishing Research Consortium
>Email:  info--publishingresearch.net
>Website:  www.publishingresearch.net

It is notoriously tricky (at least since David Hume) to "prove" causality
empirically. The thrust of the Craig et al. critique is that despite the
fact that virtually all studies comparing the citation counts for OA and
non-OA articles keep finding the OA citation counts to be higher, it has
not been proven beyond a reasonable doubt that the relationship is causal.

I agree: It is merely highly probable, not proven beyond a reasonable
doubt, that articles are more cited because they are OA, rather than
OA merely because they are more cited (or both OA and more cited merely
because of a third factor).

And I also agree that not one of the studies done so far is without some
methodological flaw that could be corrected.

But it is also highly probable that the results of the methodologically
flawless versions of all those studies will be much the same as the
results of the current studies. That's what happens when you have a
robust major effect, detected by virtually every study, and only ad hoc
methodological cavils and special pleading to rebut each of them with.

But I am sure those methodological flaws will not be corrected by these
authors, because -- OJ Simpson's "Dream Team" of Defense Attorneys comes
to mind -- Craig et al's only interest is evidently in finding flaws and
alternative explanations, not in finding out the truth -- if it goes
against their client's interests...

  Iain D.Craig: Wiley-Blackwell
  Andrew M.Plume, Mayur Amin: Elsevier
  Marie E.McVeigh, James Pringle: Thomson Scientific

Here is a preview of my rebuttal. It is mostly just common sense,
if one has no conflict of interest, hence no reason for special
pleading and strained interpretations:

(1) Research quality is a necessary, but not a sufficient condition for
citation impact: The research must also be accessible to be cited.

(2) Research accessibility is a necessary but not a sufficient condition
for citation impact: The research must also be of sufficient quality to
be cited.

(3) The OA impact effect is the finding that an article's citation counts
are positively correlated with the probability that that article has
been made OA: The more an article's citations, the more likely that that
article has been made OA.

(4) This correlation has at least three causal interpretations:

    (4a) OA articles are more likely to be cited.

    (4b) More-cited articles are more likely to be made OA.

    (4c) A third factor makes it more likely that some articles will be
    both made OA and more cited.

(5) Each of these causal interpretations is correct, and hence a
contributor to the OA impact effect:

    (5a) The better the article, the more likely it is to be cited,
    hence the more citations it gains if it is made more accessible
    (3a). (OA Article Quality Advantage, QA)

    (5b) The better the article, the more likely it is to be made OA
    (3b). (OA Article Quality Bias, QB)

    (5c) 10% of articles (and authors) receive 90% of citations. The
    authors of the better articles know they are better, and hence are
    more likely both to be cited and to make their articles OA, so as
    to maximize their visibility, accessibility and citations (3c). (OA
    Author QB and QA)

(6) In addition to QB and QA, there is an OA Early Access effect (EA):
providing access earlier increases citations.

(7) The OA citation studies have not yet isolated and estimated the
relative sizes of each of these (and other) contributing components
(OA also increases downloads, and downloads are correlated with later
citations).

(8) But the handwriting is on the wall as to the benefits of making
articles OA, for those with eyes to see, and no conflicting interests
to blind them.

I do agree completely, however, with erstwhile Princetonian Bob May's call
for "an evidence-based approach to the scholarly communications debate."

Stevan Harnad
http://www.princeton.edu/~harnad/
http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html