Davis study still lacks self-selection control group (and the sample is still small)
Ludo Waltman
ludo at LUDOWALTMAN.NL
Wed Nov 24 18:52:00 EST 2010
Dear colleagues,
Phil Davis has published interesting results on the question whether
open access leads to a citation advantage. In my view, Stevan Harnad's
criticism of Phil misses the point.
Stevan states that "with high variability you may need a large sample
to reach statistical significance, but it does not necessarily follow
that the size of the effect itself will be small". This is true.
However, in the case of Phil's results, it is not difficult to see
that the true effect size is unlikely to be of a magnitude that has
much practical significance.
Phil kindly provided me with the data of his study. Using his data,
the effect size can be estimated in a straightforward way. I define
the effect size as the average number of citations of OA articles
minus the average number of citations of non-OA articles (using a
three year time interval). I obtained a point estimate of -0.10, which
also follows directly from Phil's paper. The associated 95% confidence
interval is (-1.88, 1.67). (I calculated the confidence interval in
two different ways, namely under the assumption that the effect size
estimator follows a normal distribution and using a bootstrapping
approach that does not make any distributional assumptions. Both
approaches yielded the same confidence interval.) These calculations
provide essentially the same information as Phil's paper, but in a
slightly different way. It follows from the confidence interval that
the true effect size is unlikely to be greater than 1.67, which means
that a reasonable upper bound for the OA citation advantage in the
journals studied by Phil is about 16% (1.67 divided by 10.7, i.e.,
1.67 divided by the average number of citations of all articles in
Phil's study). Hence, even though Phil's sample is much smaller than
in some other studies, he is able to show that in the journals he
studied an OA citation advantage is either absent or of a rather small
size (especially when compared with the results reported in some other
studies).
Stevan also states that "(with a sufficiently large sample and a long
enough interval), there is (just about) always a significant, positive
OA citation advantage -- in some fields and for some articles a very
large one, but for all fields a significant, positive one." This might
well be true, but it is not relevant. With a larger sample, Phil might
have obtained a statistically significant, positive OA citation
advantage, but the effect size would still be small. Small effects,
although statistically significant, have little or no practical
significance.
In my view, Phil has convincingly shown that, at least for the
journals and the time intervals he studied, there is no meaningful OA
citation advantage.
Best regards,
Ludo Waltman
========================================================
Ludo Waltman MSc
Researcher
Centre for Science and Technology Studies
Leiden University
P.O. Box 905
2300 AX Leiden
The Netherlands
Willem Einthoven Building, Room B5-35
Tel: +31 (0)71 527 5806
Fax: +31 (0)71 527 3911
E-mail: waltmanlr at cwts.leidenuniv.nl
Homepage: www.ludowaltman.nl
========================================================
Quoting Stevan Harnad <harnad at ECS.SOTON.AC.UK>:
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> On 2010-11-24, at 9:41 AM, Philip Davis wrote:
>
>> Stevan,
>> Your new interest in sample sizes implies -- although you don't
>> seem willing to admit -- that an OA citation advantage is much,
>> much smaller than initially reported. Early studies (including
>> yours) estimated the citation effect to be somewhere between 50%
>> and 500% -- ranges that should be easily detectable with smaller
>> sample sizes such as our study. By focusing on the fact that I do
>> not have the statistical power to detect very small differences is
>> really an admission that an OA citation advantage -- if one truly
>> exists -- can be largely explained by other theories (e.g.
>> self-selection) and that the part attributable to free access is
>> very small indeed.
>
> Phil,
>
> I've always been interested in sample sizes. That's why all of our
> studies have been based on samples that have been orders of
> magnitude bigger than (for example) yours (see again Table 1, below)!
>
> But let's not confuse effect-size and the sample-size needed to
> detect a statistically significant effect; that's not a question
> about effect size but about variability: With high variability you
> may need a large sample to reach statistical significance, but it
> does not necessarily follow that the size of the effect itself will
> be small!
>
> The size of the OA citation advantage does indeed vary considerably
> (from field to field, year to year, and sample to sample) (see
> Figure 1 and Figure 2, below). The biggest effect observed has been
> in physics, where there is the added advantage of providing OA to
> the unrefereed preprint as much as a year or more prior to
> publication (but this is risky, and not advisable in all fields).
>
> But our fundamental point has been that the OA citation advantage is
> positive, significant, and present in every field tested so far --
> bigger in some fields than others, but (just about) always positive,
> significant, and there. (There have been a few other
> non-replications, usually on small samples: Your non-replication is
> not the first. Bigger samples and longer intervals make it more
> likely that you will detect the effect. See Figure 3, below.)
>
> Whenever I've written of the size of the OA citation advantage, I've
> referred to the entire observed range of the effect, not just to
> its maximum observed value, nor even just its mean, median or mode.
> But whenever I've translated the OA citation advantage into its
> potential economic benefit (e.g., how much mandating OA can enhance
> the percentage of citation impact per dollar spent on research),
> I've always used, conservatively, only the lowest end of that range.
>
> Moreover, our latest study (Gargouri et al 2010) confirmed the
> Pareto/Seglen effect, which is that not only are citations not
> equally distributed across all articles (the distribution is highly
> skewed, with the top 20% of articles receiving 80% of the
> citations), but the OA citation advantage is likewise
> correspondingly skewed (see Figure 4, below), with the most citeable
> articles benefitting the most from OA (a user self-selection
> effect, not an author self-selection effect). This variability also
> underlies the variation in the size of the OA citation advantage.
>
> So, yes, unciteable articles will not benefit from OA at all. And
> the most citeable articles will benefit most. But (with a
> sufficiently large sample and a long enough interval), there is
> (just about) always a significant, positive OA citation advantage --
> in some fields and for some articles a very large one, but for all
> fields a significant, positive one.
>
> Overall, across all fields of scientific and scholarly research
> produced by universities and funded by funders, that adds up to a
> sizeable benefit to research, researchers, their institutions, their
> funders, and the public that funds the funders and for whose
> benefit the research is being done, and funded -- a benefit that is
> worth having, by mandating OA, rather than continuing to lose,
> needlessly, as now.
>
> That implication is very clear -- and it certainly is not the
> implication you cite in your December summary in the APS house
> journal, The Physiologist:
>
> PD:
> "The fact that we observe an increase in readership and visitors for
> Open Access articles but no citation advantage suggests that
> scientific authors are adequately served by the current APS model of
> information dissemination."
>
> What your findings show is that there was no OA citation advantage
> in your (small) sample. Point taken. But the interpretation is a
> mighty stretch, if not an exercise in APS spin.
>
> There is something far, far bigger and more important at stake here
> than the revenue streams and modus operandi of APS -- or any other
> journal publisher. It's time for the publisher tail to stop trying
> to wag the research dog...
>
> Stevan
>
>
>
>
>
>
>
>
>
>
>> Stevan Harnad wrote:
>>>
>>>> Critics of our open access publishing experiment (read: Stevan
>>>> Harnad) have expressed skepticism that we were too eager to
>>>> report our findings and should have waited between 2 and 3 years.
>>>> All of the articles in our study have now aged 3-years and we
>>>> report [1] that our initial findings [2] were robust: articles
>>>> receiving the open access treatment received more article
>>>> downloads but no more citations.
>>>>
>>>> ARTICLE DOWNLOADS
>>>> During the first year of publication, open access articles
>>>> received more than double the number of full-text downloads
>>>> (119%, 95% C.I. 100% - 140%) and 61% more PDF downloads (95% C.I.
>>>> 48% - 74%) from a third more unique visitors (32%, 95% C.I. 24%
>>>> - 41%). Abstract views were reduced by nearly a third (-29%, 95%
>>>> C.I. -34% - -24%) signaling a reader preference for the full
>>>> article when available.
>>>>
>>>> ARTICLE CITATIONS
>>>> Thirty-six months after publication, open access treatment
>>>> articles were cited no more frequently than articles in the
>>>> control group (Figure 2). Open access articles received, on
>>>> average, 10.6 citations (95% C.I. 9.2 -12.0) compared to 10.7
>>>> (95% C.I. 9.6 - 11.8) for the control group. No significant
>>>> citation differences were detected at 12, 18, 24 and 30 months
>>>> after publication.
>>>>
>>>>
>>>> 1. Davis, P. M. 2010. Does Open Access Lead to Increased
>>>> Readership and Citations? A Randomized Controlled Trial of
>>>> Articles Published in APS Journals. The Physiologist 53: 197-201.
>>>>
>>>> http://www.the-aps.org/publications/tphys/2010html/December/open_access.htm
>>>>
>>>> 2. Davis, P. M., Lewenstein, B. V., Simon, D. H., Booth, J. G., &
>>>> Connolly, M. J. L. 2008. Open access publishing, article
>>>> downloads and citations: randomised trial. BMJ 337: a568.
>>>> http://dx.doi.org/10.1136/bmj.a568
>>
>>
>> --
>> Philip M. Davis, Ph.D.
>> Department of Communication
>> 301 Kennedy Hall
>> Cornell University, Ithaca, NY 14853
>> email: pmd8 at cornell.edu
>> phone: 607 255-2124
>> https://confluence.cornell.edu/display/~pmd8/resume
>> http://scholarlykitchen.sspnet.org/author/pmd8/
>
>
More information about the SIGMETRICS
mailing list