Davis study still lacks self-selection control group (and the sample is still small)

[Stevan Harnad]

On 2010-11-23, at 9:46 AM, Philip Davis wrote:

> Critics of our open access publishing experiment (read: Stevan Harnad) have expressed skepticism that we were too eager to report our findings and should have waited between 2 and 3 years.  All of the articles in our study have now aged 3-years and we report [1] that our initial findings [2] were robust: articles receiving the open access treatment received more article downloads but no more citations.
> 
> ARTICLE DOWNLOADS
> During the first year of publication, open access articles received more than double the number of full-text downloads (119%, 95% C.I. 100% - 140%) and 61% more PDF downloads (95% C.I. 48% - 74%) from a third more unique visitors (32%, 95% C.I. 24% - 41%). Abstract views were reduced by nearly a third (-29%, 95% C.I. -34% - -24%) signaling a reader preference for the full article when available.
> 
> ARTICLE CITATIONS
> Thirty-six months after publication, open access treatment articles were cited no more frequently than articles in the control group (Figure 2). Open access articles received, on average, 10.6 citations (95% C.I. 9.2 -12.0) compared to 10.7 (95% C.I. 9.6 - 11.8) for the control group. No significant citation differences were detected at 12, 18, 24 and 30 months after publication.
> 
> 
> 1. Davis, P. M. 2010. Does Open Access Lead to Increased Readership and Citations? A Randomized Controlled Trial of Articles Published in APS Journals. The Physiologist 53: 197-201. http://www.the-aps.org/publications/tphys/2010html/December/open_access.htm
> 
> 2. Davis, P. M., Lewenstein, B. V., Simon, D. H., Booth, J. G., & Connolly, M. J. L. 2008. Open access publishing, article downloads and citations: randomised trial. BMJ 337: a568. http://dx.doi.org/10.1136/bmj.a568

See:  Correlation, Causation, and the Weight of Evidence http://openaccess.eprints.org/index.php?/archives/772-guid.html

Phil Davis's dissertation results are welcome and interesting, and include some good theoretical insights, but insofar as the OA Citation Advantage is concerned, the empirical findings turn out to be just a failure to replicate the OA Citation Advantage in that particular sample and time-span. Phil's original 2008 sample of 247 OA and 1372 non-OA articles in 11 journals one year after publication has now been extended to 712 OA and 2533 non-OA articles in 36 journals 2-3 years after publication. The result is a significant download advantage for OA articles but no significant citation advantage. (Brody et al (2006) reported -- for physics article in Arxiv -- that a download advantage in the first 6 months after publication is correlated with a citation advantage 1.5 years or more after that; see also Gentil-Beccot et al's (2009) data, below). 

Brody, T., Harnad, S. and Carr, L. (2006) Earlier Web Usage Statistics as Predictors of Later Citation Impact. Journal of the American Association for Information Science and Technology (JASIST) 57(8) pp. 1060-1072. 

Gentil-Beccot, A, Mele, S  & Brooks TC (2009) Citing and reading behaviours in high-energy physics. Scientometrics 10.1007/s11192-009-0111-1 

The only way to describe this outcome is as a non-replication of the OA Citation Advantage on this particular sample; it is most definitely not a demonstration that the OA Advantage is an artifact of self-selection, since there is no control group demonstrating the presence of the citation advantage with self-selected OA and the absence of the citation advantage with randomized OA across the same sample and time-span: There is simply the failure to detect any citation advantage at all.

This failure to replicate is almost certainly due to the small sample size as well as the short time-span. (Phil's a-priori estimates of the sample size required to detect a 20% difference took no account of the fact that citations grow with time; and the a-priori criterion fails even to be met for the self-selected subsample of 65.) 

"I could not detect the effect in a much smaller and briefer sample than others" is hardly news! Compare the sample size of Phil's negative results with the sample-sizes and time-spans of some of the studies that found positive results:



And here is how the OA citation advantage builds up with time (read the curves on the left from bottom to top to see the effect of a longer and longer time interval: [the topmost curve should read "1998-2008" rather than "1998-2001"]):



Gargouri, Y., Hajjem, C., Lariviere, V., Gingras, Y., Brody, T., Carr, L. and Harnad, S. (2010) Self-Selected or Mandated, Open Access Increases Citation Impact for Higher Quality Research. PLOS ONE 10(5) e13636

Harnad, S. (2008) Davis et al's 1-year Study of Self-Selection Bias: No Self-Archiving Control, No OA Effect, No Conclusion. Open Access Archivangelism July 31 2008

Stevan Harnad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20101123/3130f4ac/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Slide0001a.gif
Type: image/gif
Size: 25600 bytes
Desc: Slide0001a.gif
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20101123/3130f4ac/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: citegrowth.gif
Type: image/gif
Size: 46072 bytes
Desc: citegrowth.gif
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20101123/3130f4ac/attachment-0001.gif>