Cancho, RFI "The variation of Zipf's law in human language " EUROPEAN PHYSICAL JOURNAL B 44 (2). MAR 2005. p.249-257

[Eugene Garfield]

RFI CANCHO : ramon at pil.phys.uniroma1.it


TITLE:          The variation of Zipf's law in human language (Article,
                English)
AUTHOR:         Cancho, RFI
SOURCE:         EUROPEAN PHYSICAL JOURNAL B 44 (2). MAR 2005. p.249-257
                SPRINGER, NEW YORK

ABSTRACT:       Words in humans follow the so-called Zipf's law. More
precisely, the word frequency spectrum follows a power function, whose
typical exponent is beta approximate to 2, but significant variations are
found. We hypothesize that the full range of variation reflects our ability
to balance the goal of communication, i.e. maximizing the information
transfer and the cost of communication, imposed by the limitations of the
human brain. We show that the higher the importance of satisfying the goal
of communication, the higher the exponent. Here, assuming that words are
used according to their meaning we explain why variation in beta should be
limited to a particular domain. From the one hand, we explain a non-trivial
lower bound at about beta = 1.6 for communication systems neglecting the
goal of the communication. From the other hand, we find a sudden divergence
of beta if a certain critical balance is crossed. At the same time a sharp
transition to maximum information transfer and unfortunately, maximum
communication cost, is found. Consistently with the upper bound of real
exponents, the maximum finite value predicted is about beta = 2.4. It is
convenient for human language not to cross the transition and remain in a
domain where maximum information transfer is high but at a reasonable cost.
Therefore, only a particular range of exponents should be found in human
speakers. The exponent beta contains information about the balance between
cost and communicative efficiency.

AUTHOR ADDRESS: RFI Cancho, Univ Roma La Sapienza, INFM Roma 1,
                Dipartimento Fis, Piazzale A Moro 5, I-00185 Rome, Italy