CWTS Journal Indicators

[Stephen J. Bensman]

Sylvan,
I am sorry for the late response, but I am just putting my life back in 
order after being almost murdered by my cat, Chico, a.k.a., Killer Kat.

The logic of complex systems seems to indicate that scientometric 
measures cannot be based on the mean like the impact factor or any 
other measure of central tendency representative of the population.  
That is what is meant by "scale free"--there is no measure of central 
tendency representative of the population.  All scientometric measures--
it seems--should be based on the the characteristics of the tail or right 
asymptote.  However, we are finding that this differs wildly depending 
on the structure of the field.  There seems to be no one shoe fits all. 

However, this does not seem to be the case empirically.  Ordinal 
rankings of journals by impact factor are remarkably stable over time.  I 
proved this in my article on Garfield and the impact factor posted on 
Gene's site (see pp. 66-68):

http://garfield.library.upenn.edu/bensman/bensmanegif22007.pdf

However, the real proof of this was done by Sasha Pudovkin in our 
article:

Bensman, Stephen J., Smolinsky, Lawrence J., and Pudovkin, Alexander 
I.  “Mean Citation Rate per Article in Mathematics Journals: Differences 
from the Scientific Model,” Journal of the American Society for 
Information Science and Technology 61 (July 2010), 1440-1463.

Pudovkin did not like my article, because I suggested that ordinal 
ranking of journals by impact factors where interval differences were 
only one thousandth had to be wildy unstable over time.  Pudovkin 
criticized me for reducing the lower impact factor rankings to 
just "statistical noise."  So I challenged him to prove otherwise by 
testing the rankings where interval separations were only in the 
thousandths  Here is the summary of his findings:

"To settle the dispute, it was decided to test the
stability of impact factor ranks at the low end of the range
for two distributions, Physics, Multidisciplinary, and Mathematics.
These two distributions were deliberately selected
because the first tested to be negative binomial, and therefore
nonrandom, whereas the second tested to binomial, and
therefore random. Pudovkin implemented the test. For his
samples he selected the 58 journals in Physics, Multidisciplinary
category with impact factors lower than 3.400 in
2008 and the 189 journals in the Mathematics category with
impact factors below 1.150 in 2008. Of the 58 Physics, Multidisciplinary,
titles, 50 were also ranked by impact factor
in 2003, and, of the 189 Mathematics journals, 143 were
also ranked by impact factor in 2003. For both sets of journals,
Pudovkin converted both their 2003 and 2008 impact
factor rankings from the ratio to the ordinal scale and calculated
their Spearman rank order correlation coefficient or
rho. The Physics, Multidisciplinary, rho was 0.86, whereas
the Mathematics rho was 0.65 and 0.68 with the elimination
of an extreme outlier. Due to these results, Pudovkin (pers.
commun. Dec. 9, 2009) declared, “Thus, your point that the
variation of IF values in the lower range of values is due only
to sampling error is certainly wrong!” He added that 5 years is
enough time for a journal to change its scope or for a science
field to change its research priorities.."  p. 1460.

I was so impressed that I asked him to be third author.

I recently refereed an article for Scientometrics, where a Spanish 
physicist proved that the third decimal place in the impact factor is 
mathematically necessary.  I just suggested that he merge his paper 
with Pudovkin's findings, but I have not heard whether he has done 
this, but I hope that he has.

>From the above--despite your complex system theories--the empirical 
evidence is that impact factor ordinal rankings are remarkably stable 
over time and therefore must be based on some powerful probability of 
being so.  This should send you back to the drawing board, where a 
recent JASIST referee sent me for my complex system musings in 
respect to Google Scholar.

Cheers to you, Sylvan,

Stephen J. Bensman, Ph.D.
LSU Libraries
Louisiana State University
Baton Rouge, LA 70803 USA
  


On Mon, 30 Sep 2013 08:43:55 -0600, Sylvan Katz 
<j.s.katz at SUSSEX.AC.UK> wrote:

>Adminstrative info for SIGMETRICS (for example unsubscribe):
>http://web.utk.edu/~gwhitney/sigmetrics.html
>
>Nees Jan
>
>> the citation distributions underlying the SNIP calculation. In general,
>> however, citation distributions do not exactly follow a power law (I 
am
>> assuming that this is what you mean by a ‘scaling distribution’),
>> although their tail may have power law properties, at least in an
>> approximate sense. Given the skewed nature of citation 
distributions, I
>
>In a scaling or power law distribution only the tail of the distribution
>exhibits a power law. The magnitude of the scaling exponent of the 
tail
>has an impact on whether or not the distribution can be characterized 
by
>its mean and variance.
>
>When the exponent is greater than or equal to 3.0 the distribution can
>be characterized by it mean and variance. However, when the 
exponent is
>less than 3.0 the variance become infinite, the central limit theorem
>(CLT) no longer applies and the distribution can no longer be
>characterized by its mean and variance. This has important implication
>for any average based measure. Newman explained this as follows in a
>sigmetrics posting two years ago
>
>https://listserv.utk.edu/cgi-bin/wa?
A2=ind1109&L=sigmetrics&T=0&F=&S=&X=1CF66970C633426B19&P=36
93
>
>"for the Central Limit Theorem to be applicable, and hence for the 
mean
>to thee valid, the distribution has to fall in the "domain of attraction
>of the Gaussian distribution".  As others have pointed out, the Pareto
>or power-law distribution to which the citation distribution is believed
>to approximate, does not fall in this domain of attraction if its
>exponent is less than 3.  Thus, the theorem is not wrong, but it's not
>applicable here."
>
>"What does this mean in practice?  Of course one can always calculate 
a
>mean number of citations for a given data sample.  But if one 
calculates
>such means for different samples -- even samples drawn from the 
exact
>same underlying distribution -- one will get wildly different answers.
>Indeed, it can be shown that the values of the mean themselves 
follow a
>power law under these circumstances, and hence can themselves vary 
over
>orders of magnitude."
>
>For a detailed explanation see Newman, M. E. J. (2005). Power laws,
>Pareto distributions and Zipf’s law. Contemporary Physics, 46(5),
>323-351. See section 3.2.
>
>For an examples of how this effects average based bibliometric
>indicators see my keynote presentation at STI 2012 "Scale 
Independent
>Measures: Theory and Practice" (
>http://sticonference.org/index.php?page=proc )
>
>> when average-based measures such as SNIP are complemented with 
stability
>> intervals, I believe that this offers a sufficiently robust approach to
>> deal with the skewed nature of citation distributions.
>
>When using average-based indicators based it is important to know 
the
>distribution of the underlying primary measures. If the distribution is
>a scaling distribution with an exponent less than 3.0 then while the
>average make be calculable it maybe meaningless as the variance 
maybe
>infinite since central limit theorem would no long applies.
>
>It seems to me that for any bibliometric indicator based on averages 
to
>be robust the underlying distributions of the primary measures need to
>be shown to fall within the Gaussian domain. And since the exponent 
of a
>citation distribution can be greater than, less than or equal to 3.0
>then the distribution likely has to be determined each time the
>indicator is calculated since in some instances the distribution will
>fall within a Gaussian distribution and at other times it may be a
>Pareto distribution with a meaningless average.
>
>It would useful to know if the SNIP indicator shows any of these
>sensitivities and hence it would be useful to know if the distribution
>of citations in a given year to papers in the preceding three years
>scales and if it does can the exponent be less than 3.0
>
>Cheers
>Sylvan