Whether Self-Selected or Mandated, Open Access Increases Citation Impact for Higher Quality Research

[Philip Davis]

Stevan,
Granted, you may be more interested in what the referees of the paper 
have to say than my comments; I'm interested in whether this paper is 
good science, whether the methodology is sound and whether you interpret 
your results properly.

For instance, it is not clear whether your Odds Ratios are interpreted 
correctly.  Based on Figure 4, OA article are MORE LIKELY to receive 
zero citations than 1-5 citations (or conversely, LESS LIKELY to receive 
1-5 citations than zero citations).

You write: "For example, we can say for the first model that for a one 
unit increase in OA, the odds of receiving 1-5 citations (versus zero 
citations) increased by a factor of 0.957. Figure 4.. (p.9)

Similarly in Figure 4 (if I understand the axes correctly), CERN article 
are more than twice as likely to be in the 20+ citation category than in 
the 1-5 citation category, a fact that may distort further 
interpretation of your data as it may be that institutional effects may 
explain your Mandated OA effect.  See comments by Patrick Gaule and Ludo 
Waltman on the review http://j.mp/8LK57u

--Phil Davis






Stevan Harnad wrote:
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> On 7-Jan-10, at 6:50 AM, Philip Davis wrote:
>
>> An interesting bit of research, although I have some methodological 
>> concerns about how you treat the data, which may explain some 
>> inconsistent and counter-intuitive results, see:
>> http://j.mp/8LK57u
>> A technical response addressing the methodology is welcome.
>> Philip M. Davis
>> PhD Student
>> Department of Communication
>> 301 Kennedy Hall
>> Cornell University, Ithaca, NY 14853
>> email: pmd8 at cornell.edu
>> phone: 607 255-2124
>> https://confluence.cornell.edu/display/~pmd8/resume
>> http://scholarlykitchen.sspnet.org/author/pmd8/
>
> Thanks for the feedback. We reply to the three points of substance, in 
> order of importance:
>
> (1) LOG RATIOS: We analyzed log citation ratios to adjust for 
> departures from normality. Logs were used to normalize the citations 
> and attenuate distortion from high values. This approach loses some 
> values when the log tranformation makes the denominator zero, but 
> despite these lost data, the t-test results were significant, and were 
> further confirmed by our second, logistic regression analysis. Moed's 
> (2007) point was about (non-log) ratios that were not used in this 
> study. We used the ratio of log citations and not the log of citation 
> ratios. When we compare log3/log2 with log30/log20, we don't compare 
> percentages with percentages (60% with 14%) because the citation 
> values are transformed or normalized: the higher the citations, the 
> stronger the normalisation. It is highly unlikely that any of this 
> would introduce a systematic bias in favor of OA, but if the referees 
> of the paper should call for a "simpler and more elegant" analysis to 
> make sure, we will be glad to perform it.
>
> (2) Effect Size: The size of the OA Advantage varies greatly from year 
> to year and field to field. We reported this in Hajjem et al (2005), 
> stressing that the important point is that there is virtually always a 
> positive OA Advantage, absent only when the  sample is too small or 
> the effect is measured too early (as in Davis et al's 2008 study). The 
> consistently bigger OA Advantage in physics (Brody & Harnad 2004) is 
> almost certainly an effect of the Early Access factor, because in 
> physics, unlike in most other disciplines (apart from computer science 
> and economics), authors tend to make their unrefereed preprints OA 
> well before publication. (This too might be a good practice to 
> emulate, for authors desirous of greater research impact.)
>
> (3) Mandated OA Advantage? Yes, the fact that the citation advantage 
> of mandated OA was slightly greater than that of self-selected OA is 
> surprising, and if it proves reliable, it is interesting and worthy of 
> interpretation. We did not interpret it in our paper, because it was 
> the smallest effect, and our focus was on testing the 
> Self-Selection/Quality-Bias  hypothesis, according to which mandated 
> OA should have little or no citation advantage at all, if 
> self-selection is a major contributor to the OA citation advantage.
>
> Our sample was 2002-2006. We are now analyzing 2007-2008. If there is 
> still a statistically significant OA advantage for mandated OA over 
> self-selected OA in this more recent sample too, a potential 
> explanation is the inverse of the Self-Selection/Quality-Bias 
> hypothesis (which, by the way, we do think is one of the several 
> factors  that contribute to the OA Advantage, alongside the other 
> contributors:  Early Advantage, Quality Advantage, Competitive 
> Advantage, Download Advantage, Arxiv Advantage, and probably others).  
> http://openaccess.eprints.org/index.php?/archives/29-guid.html
>
> The Self-Selection/Quality-Bias (SSQB) consists of better authors 
> being more likely to make their papers OA, and/or authors being more 
> likely to make their better papers OA, because they are better, hence 
> more citeable. The hypothesis we tested was that all or most of the 
> widely reported OA Advantage across all fields and years is just due 
> to SSQB. Our data show that it is not, because the OA Advantage is no 
> smaller when it is mandated. If it turns out to be reliably bigger, 
> the most likely explanation is a variant of the "Sitting Pretty" (SP) 
> effect, whereby some of the more comfortable authors have said that 
> the reason they do not make their articles OA is that they think they 
> have enough access and impact already. Such authors do not 
> self-archive spontaneously. But when OA is mandated, their papers reap 
> the extra benefit of OA, with its Quality Advantage (for the better, 
> more citeable papers). In other words, if SSQB is a bias in favor of 
> OA on the part of some of the better authors, mandates reverse an SP 
> bias against OA on the part of others of the better authors. 
> Spontaneous, unmandated OA would be missing the papers of these SP 
> authors. http://www.eprints.org/openaccess/self-faq/#29.Sitting
>
> There may be other explanations too. But we think any explanation at 
> all is premature until it is confirmed that this new mandated OA 
> advantage is indeed reliable and replicable. Phil further singles out 
> the fact that the mandate advantage is present in the middle citation 
> ranges and not the top and bottom. Again, it seems premature to 
> interpret these minor effects whose unreliability is unknown, but if 
> forced to pick an interpretation now, we would say it was because the 
> "Sitting Pretty" authors may be the middle-range authors rather than 
> the top ones...
>
> Yassine Gargouri, Chawki Hajjem, Vincent Lariviere, Yves Gingras, Les 
> Carr, Tim Brody, Stevan Harnad
>


-- 
Philip M. Davis
PhD Student
Department of Communication
301 Kennedy Hall
Cornell University, Ithaca, NY 14853
email: pmd8 at cornell.edu
phone: 607 255-2124
https://confluence.cornell.edu/display/~pmd8/resume
http://scholarlykitchen.sspnet.org/author/pmd8/