Open Access: sample size, generalizability and self-selection

[Yassine Gargouri]

(Apologies. Re-posting with references and URLs added.)

Our previous sample [1] comparing self-selective self-archiving with
mandatory self-archiving (27,197 articles from the publication interval 2002
to 2006 – 6,215 mandated and 20,982 nonmandated) has now been extended to
63,518 articles (13,425 mandated and 50,093 nonmandated) published between
2002 and 2009 in 5,992 journals. For all OA vs Non-OA (O/Ø) comparisons,
regardless of whether the OA was Self-Selected (S) or Mandated (M), the mean
log citation differences (after adding a constant value 1 to all citations
in order to include uncited papers) are significantly greater than zero
(based on correlated-sample 2-tailed t-tests for within-journal differences
(p = 0.05).

The t-tests applied on the 7 post hoc differences showed in this table,
averaged across 2004-2009 (because mandates began to be adopted in 2004)
have a statistical power of about 100% (except for the last difference OM vs
OS, which is only 11%, and hence we discounted it in our interpretation).

Based on the same means, standard deviations and correlation coefficients as
for the first pair of comparisons (O vs Ø) of 3,578 journals, the a priori
estimate of statistical power shrinks to 23% when the sample of journals is
reduced to 36 (as in Davis’s sample). A minimum sample size of 183 journals
is required to get a significant effect. 

Davis’s study [2, 3] seems to have calculated the minimum sample size needed
in order to reach a relative statistical power of 80% in terms of the number
of articles within each journal, but not in terms of the number of journals
(36).

It follows that a failure to replicate the OA citation advantage with such a
sample size would not be improbable.


References

1. Gargouri Y, Hajjem C, Larivière V, Gingras Y, Carr L, Brody T, Harnad S
(2010) Self-Selected or Mandated, Open Access Increases Citation Impact for
Higher Quality Research. PLoS ONE 5(10):e13636+.
doi:10.1371/journal.pone.0013636.
http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0013636

2. Davis, P. M. (2010) Does Open Access Lead to Increased Readership and
Citations? A Randomized Controlled Trial of Articles Published in APS
Journals. The Physiologist 53(6): 197-201.
http://www.the-aps.org/publications/tphys/2010html/December/open_access.htm

3. Davis P.M. (2010) Access, readership, citations: a randomized controlled
trial of scientific journal publishing (PhD dissertation). Ithaca: Cornell
University.
http://ecommons.cornell.edu/bitstream/1813/17788/1/Davis%2c%20Philip.pdf


-------------- next part --------------
A non-text attachment was scrubbed...
Name: table.jpg
Type: image/jpeg
Size: 68638 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20101221/b243817b/attachment.jpg>