Open Access: sample size, generalizability and self-selection

[Philip Davis]

Stevan,
You seem to have conceded that your small sample size critique does not 
hold.  Let's move to your new concern about generalizability:

While I can't claim negative results across all fields and across all 
times, our randomized controlled trials (RCTs) did involve 36 journals 
produced by 7 different publishers in the medical, biological, and 
multi-disciplinary sciences, plus the social sciences and humanities.  
The nature of the RCTs means a lot of human intervention goes in to set 
up and run the experiments.  In comparison, retrospective observational 
studies (the studies you cite as comparisons) are largely automated and 
are able to gather a huge amount of data quickly with little human 
intervention.  Yet, if you are basing your comparison solely on number 
of journals and number of articles, then you are completely missing the 
rationale for conducting the RCTs in the first place:

By design, RCTs are better at isolating possible causes, determining the 
direction of causation, and ruling out confounding variables.  While it 
is impossible to prove cause and effect, RCTs generally provide much 
stronger evidence than retrospective observational studies.

Your last concern was about a self-selection control group:

There were not a lot of cases of self-archiving in our dataset.  
Remember that we were not studying physics and that our studies began in 
2007.  You will note that I report a positive citation effect in my 
Appendix, but because the act of self-archiving was out of our control, 
we could not distinguish between access and self-selection as a 
definitive cause.  I also report in my dissertation that articles 
selected and promoted by editors were more highly-cited, but it appears 
that editors were simply selecting more citable articles (e.g. reviews) 
to promote.


--Phil



-- 
Philip M. Davis, Ph.D.
Department of Communication
301 Kennedy Hall
Cornell University, Ithaca, NY 14853
email: pmd8 at cornell.edu
phone: 607 255-2124
https://confluence.cornell.edu/display/~pmd8/resume
http://scholarlykitchen.sspnet.org/author/pmd8/