Open Access: sample size, generalizability and self-selection

[Yassine Gargouri]

Our previous sample comparing self-selective self-archiving with mandatory
self-archiving (27,197 articles from the publication interval 2002 to 2006 –
6,215 mandated and 20,982 nonmandated) has now been extended to 63,518
articles (13,425 mandated and 50,093 nonmandated) published between 2002 and
2009 in 5,992 journals. For all OA vs Non-OA (O/Ø) comparisons, regardless
of whether the OA was Self-Selected (S) or Mandated (M), the mean log
citation differences (after adding a constant value 1 to all citations in
order to include uncited papers) are significantly greater than zero (based
on correlated-sample 2-tailed t-tests for within-journal differences (p = 0.05).

The t-tests applied on the 7 post hoc differences showed in this table,
averaged across 2004-2009 (because mandates began to be adopted in 2004)
have a statistical power of about 100% (except for the last difference OM vs
OS, which is only 11%, and hence we discounted it in our interpretation).

Based on the same means, standard deviations and correlation coefficients as
for the first pair of comparisons (O vs Ø) of 3,578 journals, the a priori
estimate of statistical power shrinks to 23% when the sample of journals is
reduced to 36 (as in Davis’s sample). A minimum sample size of 183 journals
is required to get a significant effect. 

Davis’s study seems to have calculated the minimum sample size needed in
order to reach a relative statistical power of 80% in terms of the number of
articles within each journal, but not in terms of the number of journals (36).

It follows that a failure to replicate the OA citation advantage with such a
sample size would not be improbable.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: table.jpg
Type: image/jpeg
Size: 68638 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20101221/58942395/attachment.jpg>