van den Besselaar "Empirical evidence of self-organization?" JASIST 54(1):87-90, 2003

[Loet Leydesdorff]

Dear Gene:

Unfortunately, my reply could not be published as a rejoinder, but in
the meantime it was accepted as a Letter to the Editor of JASIST. I take
the liberty to provide the preprint version below, since it is
relatively short.

With kind regards, Loet

====================================================================
Letter to the Editor

In his paper entitled "Empirical evidence of self-organization?," Peter
van den Besselaar argued that the results of Leydesdorff & Heimeriks
(2001) are based on unreliable statistics. Two arguments were brought
against the use of discriminant analysis in our case:

1.      The predictive power of 50% of the sample in relation to the
other 50% is not significant;
2.      Using simulations, discriminant analysis exhibits a prediction
for random data that can be as high as for the bibliometric data.

ad 1. Dividing the sample

In the case of different samples which are representative of a
population, one can use the eigenstructure of the first 50% for the
prediction of the eigenstructure in the other 50% as a test. However, we
were studying journal publications which are not representative of a
population, but selected according to very specific criteria. The
eigenstructures of these selections are sample-specific.

In a selected sample, the prediction can be expected to deteriorate
rapidly as one extends beyond the original sample by adding new cases.
The negative results reported by Van den Besselaar (2002) are therefore
not surprising. His test does not work because one cannot expect any
significant correlation between the eigenstructures of highly specific
samples.

To pursue this line of reasoning, a better strategy might have been to
proceed stepwise by iteratively recalculating the eigenstructure after
each addition of a new case. (I have suggested this heuristic to Van den
Besselaar in a previous exchange.)

ad 2. The simulations

Van den Besselaar claims that our results are also invalid because we
disregarded his simulation results. This is simply not true: we were
thoroughly familiar with his results when writing our paper. Our paper
contains a reference to Van den Besselaar & Heimeriks (2000) that was
published first. Furthermore, we replicated the simulations and found
precisely the same results. These results were the sole ground for
making the inference on p. 1266 of our paper that "(...) the weak
structure in the network of words does also not significantly correlate
with the geographical division."

The second-order analysis of "self-organization" that follows after this
conclusion was therefore not based on the results of the first-order
discriminant analysis, as the reader can easily check. Van den Besselaar
has misread and selectively quoted our paper. We repeated the argument
that we had discarded the first-order results in the conclusion section
(at p. 1272):

        "The two types of analysis are of a different nature. Whereas no
statistical significance could be retrieved inductively in the initial
analysis of word patterns, these negative results did not prevent us
from using word patterns in testing the second-order hypothesis. (...)
(S)econd-order theorizing has a more elusive character, because one
proceeds on the basis of hypothetical "what-if"-types of questions."

We used the results of the discriminant analysis only in order to remove
records that had initially been flagged as misplaced. In our opinion,
this deletion was prudent because the analytical causes of the
simulation results were not clear. In my opinion, the simulated data are
also different from the real ones. For example, the simulation results
usually did not pass the significance tests provided by SPSS (because
they were based on randomness), while our results using bibliometric
data did pass these tests.

Conclusion

The focus of our paper was not on data collection, but on developing a
new methodology. However, our results are not invalid in terms of the
statistics used for the two reasons mentioned by Van den Besselaar. His
first argument is not applicable to our case, and his second argument
was used by us as a reason for moving from a first-order to a
second-order analysis.


References:

Leydesdorff, L., & Heimeriks, G. (2001). The Self-Organization of the
European Information Society: The Case of "Biotechnology". Journal of
the American Society for Information Science & Technology 52(14),
1262-1274.

Van den Besselaar, P. (2002). Empirical evidence of self-organization?
Journal of the American Society for Information Science & Technology
54(1), 87-90.

Van den Besselaar, P., & Heimeriks, G. (2000). Codification and
self-organization in the European STI system. Final report to the
European Commission. Amsterdam: University of Amsterdam.

  _____

Loet Leydesdorff
Science & Technology Dynamics, University of Amsterdam
http://www.leydesdorff.net/


> -----Original Message-----
> From: ASIS&T Special Interest Group on Metrics
> [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Eugene Garfield
> Sent: Friday, January 24, 2003 9:17 PM
> To: SIGMETRICS at LISTSERV.UTK.EDU
> Subject: [SIGMETRICS] van den Besselaar "Empirical evidence
> of self-organization?" JASIST 54(1):87-90, 2003
>
>
> Peter van dan Besselaar :   peter.van.den.besselaar at niwi.knaw.nl
>
>
> Title       Empirical evidence of self-organization?
> Author      van den Besselaar P
> Journal     JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION
>             SCIENCE AND TECHNOLOGY  54 (1): 87-90 JAN 1 2003
>
>  Document type: Article  Language: English
>  Cited References: 6     Times Cited: 0
>
>
> Abstract:
> In a recent paper in this journal, Loet Leydesdorff and
> Gaston Heimeriks (2001, Journal of the American Society for
> Information Science and Technology, 52, 1262-1294.) argue
> that biotechnology develops in a self-organizational mode,
> through interaction between the intellectual structure and
> the institutional network of  the research field. This claim
> is empirically supported by a multivariate analysis of
> documents from core biotechnology journals. One unexpected
> finding in this paper is the relationship between the title
> words of documents and the region of their origin. This claim
> requires examination because, as will be shown, it seems to
> be an artifact of the method used. If this is so, it
> undermines the authors' theoretical claim that the production
> of knowledge is a self-organizing process.
>
> Addresses:
> van den Besselaar P, Royal Netherlands Acad Arts & Sci NIWI
> KNAW, Dept Social Sci, POB 95110, NL-1000 HC Amsterdam,
> Netherlands Royal Netherlands Acad Arts & Sci NIWI KNAW, Dept
> Social Sci, NL-1000 HC Amsterdam, Netherlands
>
> Publisher:
> JOHN WILEY & SONS INC, HOBOKEN
>
> IDS Number:
> 626ZQ
>
> ISSN:
> 1532-2882
>
>
> Cited Author            Cited Work                Volume
> Page   Year
>
>  BATTACHARYA S         SCIENTOMETRICS                43
> 359    1998
>  KLECKA WR             DISCRIMINANT ANAL
>        1980
>  LEYDESDORFF L         J AM SOC INF SCI TEC          52
> 1262    2001
>  LEYDESDORFF L         J AM SOC INFORM SCI           48
> 418    1997
>  NORUSIS MJ            SPSS PC PLUS PROFESS
>        1992
>  VANDENBESSELAAR P     SCIENTOMETRICS                47
> 169    2000
>