Sigurd, B; Eeg-Olofsson, M; van de Weijer, J "Word length, sentence length and frequency - Zipf revisited" STUDIA LINGUISTICA 58 (1). APR 2004. p.37-52 BLACKWELL PUBL LTD, OXFORD

[Eugene Garfield]

B. Sigurd       : Bengt.Sigurd at ling.lu.se
M. Eeg-Olofsson : Mats.Eeg-Olofsson at ling.lu.se
J. van de Weijer: vdweijer at ling.lu.se

TITLE:          Word length, sentence length and frequency - Zipf
                revisited (Article, English)
AUTHOR:         Sigurd, B; Eeg-Olofsson, M; van de Weijer, J
SOURCE:         STUDIA LINGUISTICA 58 (1). APR 2004. p.37-52 BLACKWELL
                PUBL LTD, OXFORD

SEARCH TERM(S):  ZIPF*  item_title


ABSTRACT:       This paper examines data from English, Swedish and German
in order to find a theoretical distribution that describes the observed
relation between word length and frequency. In Swedish and English, most
word tokens consist of three letters only, while shorter or longer words
occur less frequently. We found that the equation with the general form
f(exp) = a * L-b * c(L) (a variant of the so-called gamma distribution)
approximates the observed frequencies reasonably well. This formula
incorporates both the fact that the number of possible words increases
with word length, and the fact that longer words tend to be avoided,
presumably because they are uneconomic. To our knowledge this formula has
not been proposed to describe word frequency data. We examined frequency
distributions of word length in Swedish and English, and explored
different variants of the equation by systematically varying the a, b and
c parameters. Subsequently, we also applied the formula to the frequency
distribution of sentence length in English, and found an almost perfect
fit for a corpus consisting of different text genres. Moreover, the data
showed that the formula can be used to distinguish between different
kinds of text genres.

AUTHOR ADDRESS: B Sigurd, Lund Univ, Dept Phonet & Linguist, Helgonabacken
                12, S-22362 Lund, Sweden