CWTS Journal Indicators
Stephen J. Bensman
notsjb at LSU.EDU
Tue Oct 8 10:09:12 EDT 2013
I am sorry for the late response, but I am just putting my life back in
order after being almost murdered by my cat, Chico, a.k.a., Killer Kat.
The logic of complex systems seems to indicate that scientometric
measures cannot be based on the mean like the impact factor or any
other measure of central tendency representative of the population.
That is what is meant by "scale free"--there is no measure of central
tendency representative of the population. All scientometric measures--
it seems--should be based on the the characteristics of the tail or right
asymptote. However, we are finding that this differs wildly depending
on the structure of the field. There seems to be no one shoe fits all.
However, this does not seem to be the case empirically. Ordinal
rankings of journals by impact factor are remarkably stable over time. I
proved this in my article on Garfield and the impact factor posted on
Gene's site (see pp. 66-68):
However, the real proof of this was done by Sasha Pudovkin in our
Bensman, Stephen J., Smolinsky, Lawrence J., and Pudovkin, Alexander
I. “Mean Citation Rate per Article in Mathematics Journals: Differences
from the Scientific Model,” Journal of the American Society for
Information Science and Technology 61 (July 2010), 1440-1463.
Pudovkin did not like my article, because I suggested that ordinal
ranking of journals by impact factors where interval differences were
only one thousandth had to be wildy unstable over time. Pudovkin
criticized me for reducing the lower impact factor rankings to
just "statistical noise." So I challenged him to prove otherwise by
testing the rankings where interval separations were only in the
thousandths Here is the summary of his findings:
"To settle the dispute, it was decided to test the
stability of impact factor ranks at the low end of the range
for two distributions, Physics, Multidisciplinary, and Mathematics.
These two distributions were deliberately selected
because the first tested to be negative binomial, and therefore
nonrandom, whereas the second tested to binomial, and
therefore random. Pudovkin implemented the test. For his
samples he selected the 58 journals in Physics, Multidisciplinary
category with impact factors lower than 3.400 in
2008 and the 189 journals in the Mathematics category with
impact factors below 1.150 in 2008. Of the 58 Physics, Multidisciplinary,
titles, 50 were also ranked by impact factor
in 2003, and, of the 189 Mathematics journals, 143 were
also ranked by impact factor in 2003. For both sets of journals,
Pudovkin converted both their 2003 and 2008 impact
factor rankings from the ratio to the ordinal scale and calculated
their Spearman rank order correlation coefficient or
rho. The Physics, Multidisciplinary, rho was 0.86, whereas
the Mathematics rho was 0.65 and 0.68 with the elimination
of an extreme outlier. Due to these results, Pudovkin (pers.
commun. Dec. 9, 2009) declared, “Thus, your point that the
variation of IF values in the lower range of values is due only
to sampling error is certainly wrong!” He added that 5 years is
enough time for a journal to change its scope or for a science
field to change its research priorities.." p. 1460.
I was so impressed that I asked him to be third author.
I recently refereed an article for Scientometrics, where a Spanish
physicist proved that the third decimal place in the impact factor is
mathematically necessary. I just suggested that he merge his paper
with Pudovkin's findings, but I have not heard whether he has done
this, but I hope that he has.
>From the above--despite your complex system theories--the empirical
evidence is that impact factor ordinal rankings are remarkably stable
over time and therefore must be based on some powerful probability of
being so. This should send you back to the drawing board, where a
recent JASIST referee sent me for my complex system musings in
respect to Google Scholar.
Cheers to you, Sylvan,
Stephen J. Bensman, Ph.D.
Louisiana State University
Baton Rouge, LA 70803 USA
On Mon, 30 Sep 2013 08:43:55 -0600, Sylvan Katz
<j.s.katz at SUSSEX.AC.UK> wrote:
>Adminstrative info for SIGMETRICS (for example unsubscribe):
>> the citation distributions underlying the SNIP calculation. In general,
>> however, citation distributions do not exactly follow a power law (I
>> assuming that this is what you mean by a ‘scaling distribution’),
>> although their tail may have power law properties, at least in an
>> approximate sense. Given the skewed nature of citation
>In a scaling or power law distribution only the tail of the distribution
>exhibits a power law. The magnitude of the scaling exponent of the
>has an impact on whether or not the distribution can be characterized
>its mean and variance.
>When the exponent is greater than or equal to 3.0 the distribution can
>be characterized by it mean and variance. However, when the
>less than 3.0 the variance become infinite, the central limit theorem
>(CLT) no longer applies and the distribution can no longer be
>characterized by its mean and variance. This has important implication
>for any average based measure. Newman explained this as follows in a
>sigmetrics posting two years ago
>"for the Central Limit Theorem to be applicable, and hence for the
>to thee valid, the distribution has to fall in the "domain of attraction
>of the Gaussian distribution". As others have pointed out, the Pareto
>or power-law distribution to which the citation distribution is believed
>to approximate, does not fall in this domain of attraction if its
>exponent is less than 3. Thus, the theorem is not wrong, but it's not
>"What does this mean in practice? Of course one can always calculate
>mean number of citations for a given data sample. But if one
>such means for different samples -- even samples drawn from the
>same underlying distribution -- one will get wildly different answers.
>Indeed, it can be shown that the values of the mean themselves
>power law under these circumstances, and hence can themselves vary
>orders of magnitude."
>For a detailed explanation see Newman, M. E. J. (2005). Power laws,
>Pareto distributions and Zipf’s law. Contemporary Physics, 46(5),
>323-351. See section 3.2.
>For an examples of how this effects average based bibliometric
>indicators see my keynote presentation at STI 2012 "Scale
>Measures: Theory and Practice" (
>> when average-based measures such as SNIP are complemented with
>> intervals, I believe that this offers a sufficiently robust approach to
>> deal with the skewed nature of citation distributions.
>When using average-based indicators based it is important to know
>distribution of the underlying primary measures. If the distribution is
>a scaling distribution with an exponent less than 3.0 then while the
>average make be calculable it maybe meaningless as the variance
>infinite since central limit theorem would no long applies.
>It seems to me that for any bibliometric indicator based on averages
>be robust the underlying distributions of the primary measures need to
>be shown to fall within the Gaussian domain. And since the exponent
>citation distribution can be greater than, less than or equal to 3.0
>then the distribution likely has to be determined each time the
>indicator is calculated since in some instances the distribution will
>fall within a Gaussian distribution and at other times it may be a
>Pareto distribution with a meaningless average.
>It would useful to know if the SNIP indicator shows any of these
>sensitivities and hence it would be useful to know if the distribution
>of citations in a given year to papers in the preceding three years
>scales and if it does can the exponent be less than 3.0
More information about the SIGMETRICS