Davis study still lacks self-selection control group (and the sample is still small)

[Stevan Harnad]

On Thu, Nov 25, 2010 at 6:30 PM, Ludo Waltman <ludo at ludowaltman.nl> wrote


There is an essential difference between what Stevan Harnad says and what I
> am saying. Stevan states that the results of Phil Davis can be interpreted
> as "simply the failure to detect any citation advantage at all". He also
> states that "this failure to replicate is almost certainly due to the small
> sample size as well as the short time-span". We seem to agree that no OA
> citation advantage can be detected in Phil's data. What we do not agree
> about is whether or not this is likely to be caused by a small sample size.
> As indicated in my previous post, I consider this to be unlikely. Phil's
> results show not only that there is no OA citation advantage in his sample,
> but also that an OA citation advantage of reasonable size is unlikely to
> exist in the underlying population.
>

It depends on what you mean by "the underlying population." If you mean the
3 -year population of articles in the 36 (mostly APS) journals that Phil
actually tested, I agree.

But if you mean that general population of
articles<http://opcit.eprints.org/oacitation-biblio.html>,
in multiple different fields, tested by multiple independent investigators,
reporting a significant (and sometimes quite sizeable) OA citation advantage
(with the exception of a very small number of negative or null outcomes),
then Phil's study certainly has *not* shown "that an OA citation advantage
of reasonable size is unlikely to exist" in *that* underlying population. It
would take a null
meta-analysis<http://openaccess.eprints.org/index.php?/archives/718-guid.html>,
not just one null outcome, to be able to show that. (Otherwise any
repeatedly observed effect could be dismissed on the basis of one
non-replication!)

What Phil set out to show was that *the repeatedly observed OA citation
advantage was an artifact of author self-selection*, and the way he hoped to
show that -- and it's a good way -- was to show that randomly imposed OA
 eliminates the OA citation advantage. So far so good. But what Phil failed
to do was to replicate the OA citation advantage in the case of author
self-selection, and then show that it is eliminated by randomization -- in
the same journal population and time interval.

Instead he just found no OA citation advantage at all, for that journal
population and interval.

That, to repeat, is one non-replication, for that journal population and
time interval. It is not a demonstration that the repeatedly observed OA
citation advantage is an artifact of author self-selection.

And -- not to focus only on negative results -- let me suggest that it is
pertinent to this question that we too did a test of the self-selection
hypothesis <http://dx.plos.org/10.1371/journal.pone.0013636> -- on a much
larger sample across more fields and a longer time interval -- and we not
only found "an OA citation advantage of reasonable size" for self-selected
OA, but we found that the advantage was the same size for mandated OA.

We accordingly conclude that "an OA citation advantage of reasonable size is
likely to exist in the underlying population" if you test for it, and your
sample is big enough and long enough -- except perhaps in Phil's sample of
(mostly APS) journals...

Stevan Harnad


>
>> On Wed, Nov 24, 2010 at 11:52 PM, Ludo Waltman <ludo at ludowaltman.nl>
>> wrote:
>>
>> LW:
>>
>> Phil Davis has published interesting results on the question whether open
>>
>>> access leads to a citation advantage. In my view, Stevan Harnad's
>>> criticism
>>> of Phil misses the point...
>>>
>>
>> In my view, Phil has convincingly shown that, at least for the journals
>> and
>>
>>> the time intervals he studied, there is no meaningful OA citation
>>> advantage.
>>>
>>
>>
>> I don't understand Ludo Waltman's point, since this is exactly what I
>> said:
>>
>> SH:
>>
>> Phil Davis's dissertation results are welcome and interesting, and include
>>
>>> some good theoretical insights, but insofar as the OA Citation Advantage
>>> is
>>> concerned, the empirical findings turn out to be just a failure to
>>> replicate
>>> the OA Citation Advantage in that particular sample and time-span... it
>>> is
>>> most definitely not a demonstration that the OA Advantage is an artifact
>>> of
>>> self-selection, since there is no control group demonstrating the
>>> presence
>>> of the citation advantage with self-selected OA and the absence of the
>>> citation advantage with randomized OA across the same sample and
>>> time-span:
>>> There is simply the failure to detect any citation advantage at all.
>>>
>>
>>
>> Stevan Harnad
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20101125/0721e3c0/attachment.html>