Davis study still lacks self-selection control group (and the sample is still small)

Wed Nov 24 14:25:32 EST 2010

On 2010-11-24, at 9:41 AM, Philip Davis wrote:

> Stevan,
> Your new interest in sample sizes implies -- although you don't seem willing to admit -- that an OA citation advantage is much, much smaller than initially reported.  Early studies (including yours) estimated the citation effect to be somewhere between 50% and 500% -- ranges that should be easily detectable with smaller sample sizes such as our study.  By focusing on the fact that I do not have the statistical power to detect very small differences is really an admission that an OA citation advantage -- if one truly exists -- can be largely explained by other theories (e.g. self-selection) and that the part attributable to free access is very small indeed.

Phil,

I've always been interested in sample sizes. That's why all of our studies have been based on samples that have been orders of magnitude bigger than (for example) yours (see again Table 1, below)! 

But let's not confuse effect-size and the sample-size needed to detect a statistically significant effect; that's not a question about effect size but about variability: With high variability you may need a large sample to reach statistical significance, but it does not necessarily follow that the size of the effect itself will be small!

The size of the OA citation advantage does indeed vary considerably (from field to field, year to year, and sample to sample) (see Figure 1 and Figure 2, below). The biggest effect observed has been in physics, where there is the added advantage of providing OA to the unrefereed preprint as much as a year or more prior to publication (but this is risky, and not advisable in all fields).

But our fundamental point has been that the OA citation advantage is positive, significant, and present in every field tested so far -- bigger in some fields than others, but (just about) always positive, significant, and there. (There have been a few other non-replications, usually on small samples: Your non-replication is not the first. Bigger samples and longer intervals make it more likely that you will detect the effect. See Figure 3, below.)

Whenever I've written of the size of the OA citation advantage, I've referred to the entire observed range of the effect, not just to its maximum observed value, nor even just its mean, median or mode. But whenever I've translated the OA citation advantage into its potential economic benefit (e.g., how much mandating OA can enhance the percentage of citation impact per dollar spent on research), I've always used, conservatively, only the lowest end of that range.

Moreover, our latest study (Gargouri et al 2010) confirmed the Pareto/Seglen effect, which is that not only are citations not equally distributed across all articles (the distribution is highly skewed, with the top 20% of articles receiving 80% of the citations), but the OA citation advantage is likewise correspondingly skewed (see Figure 4, below), with the most citeable articles benefitting the most from OA (a user self-selection effect, not an author self-selection effect). This variability also underlies the variation in the size of the OA citation advantage.

So, yes, unciteable articles will not benefit from OA at all. And the most citeable articles will benefit most. But (with a sufficiently large sample and a long enough interval), there is (just about) always a significant, positive OA citation advantage -- in some fields and for some articles a very large one, but for all fields a significant, positive one. 

Overall, across all fields of scientific and scholarly research produced by universities and funded by funders, that adds up to a sizeable benefit to research, researchers, their institutions, their funders, and the public that funds the funders and for whose benefit the research is being done, and funded -- a benefit that is worth having, by mandating OA, rather than continuing to lose, needlessly, as now.

That implication is very clear -- and it certainly is not the implication you cite in your December summary in the APS house journal, The Physiologist: 

PD:
"The fact that we observe an increase in readership and visitors for Open Access articles but no citation advantage suggests that scientific authors are adequately served by the current APS model of information dissemination."

What your findings show is that there was no OA citation advantage in your (small) sample. Point taken. But the interpretation is a mighty stretch, if not an exercise in APS spin. 

There is something far, far bigger and more important at stake here than the revenue streams and modus operandi of APS -- or any other journal publisher. It's time for the publisher tail to stop trying to wag the research dog...

Stevan

> Stevan Harnad wrote:
>> 
>>> Critics of our open access publishing experiment (read: Stevan Harnad) have expressed skepticism that we were too eager to report our findings and should have waited between 2 and 3 years.  All of the articles in our study have now aged 3-years and we report [1] that our initial findings [2] were robust: articles receiving the open access treatment received more article downloads but no more citations.
>>> 
>>> ARTICLE DOWNLOADS
>>> During the first year of publication, open access articles received more than double the number of full-text downloads (119%, 95% C.I. 100% - 140%) and 61% more PDF downloads (95% C.I. 48% - 74%) from a third more unique visitors (32%, 95% C.I. 24% - 41%). Abstract views were reduced by nearly a third (-29%, 95% C.I. -34% - -24%) signaling a reader preference for the full article when available.
>>> 
>>> ARTICLE CITATIONS
>>> Thirty-six months after publication, open access treatment articles were cited no more frequently than articles in the control group (Figure 2). Open access articles received, on average, 10.6 citations (95% C.I. 9.2 -12.0) compared to 10.7 (95% C.I. 9.6 - 11.8) for the control group. No significant citation differences were detected at 12, 18, 24 and 30 months after publication.
>>> 
>>> 
>>> 1. Davis, P. M. 2010. Does Open Access Lead to Increased Readership and Citations? A Randomized Controlled Trial of Articles Published in APS Journals. The Physiologist 53: 197-201. http://www.the-aps.org/publications/tphys/2010html/December/open_access.htm
>>> 
>>> 2. Davis, P. M., Lewenstein, B. V., Simon, D. H., Booth, J. G., & Connolly, M. J. L. 2008. Open access publishing, article downloads and citations: randomised trial. BMJ 337: a568. http://dx.doi.org/10.1136/bmj.a568
> 
> 
> -- 
> Philip M. Davis, Ph.D.
> Department of Communication
> 301 Kennedy Hall
> Cornell University, Ithaca, NY 14853
> email: pmd8 at cornell.edu
> phone: 607 255-2124
> https://confluence.cornell.edu/display/~pmd8/resume
> http://scholarlykitchen.sspnet.org/author/pmd8/ 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20101124/b2a86c36/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Slide0001a.gif
Type: image/gif
Size: 25600 bytes
Desc: Slide0001a.gif
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20101124/b2a86c36/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-6.tiff
Type: image/tiff
Size: 390350 bytes
Desc: PastedGraphic-6.tiff
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20101124/b2a86c36/attachment.tiff>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-5.tiff
Type: image/tiff
Size: 876382 bytes
Desc: PastedGraphic-5.tiff
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20101124/b2a86c36/attachment-0001.tiff>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-7.tiff
Type: image/tiff
Size: 876382 bytes
Desc: PastedGraphic-7.tiff
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20101124/b2a86c36/attachment-0002.tiff>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-8.tiff
Type: image/tiff
Size: 876382 bytes
Desc: PastedGraphic-8.tiff
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20101124/b2a86c36/attachment-0003.tiff>