Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1 9 Jan 2006

Mon Mar 6 16:12:54 EST 2006

Dear Stephen,

I apologize if I dragged you into the discussion with improper argument, but
I did not want to mention your idea of using total cites as an indicator
without providing a proper reference.

The reasoning in your posting is difficult for me to follow, but I look
forward to reading the full paper. My experience is that reading the full
paper, one begins to understand. I found your previous argument about using
total cites very convincing because of its high correlation with faculty
ratings and its orthogonality to the impact factor. It seemed to me that the
impact factor measures something very different from the prestige of a
journal.

Figure 1: Component plot in rotated space (sources: JCR, 1993; Bensman,
2001; forthcoming; Bensman & Wilder, 1998).

From: Visualization of the Citation Impact
<http://www.leydesdorff.net/jcr04> Environments of Scientific Journals: An
online mapping exercise, Journal of the American Society for Information
Science and Technology (forthcoming). . <pdf-version
<http://www.leydesdorff.net/jcr04/jcr2pajek.pdf> >

With best wishes,

Loet

________________________________
Loet Leydesdorff
Amsterdam School of Communications Research (ASCoR),
Kloveniersburgwal 48, 1012 CX Amsterdam.
Tel.: +31-20- 525 6598; fax: +31-20- 525 3681;
loet at leydesdorff.net ; http://www.leydesdorff.net/

> -----Original Message-----
> From: ASIS&T Special Interest Group on Metrics
> [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman
> Sent: Monday, March 06, 2006 6:41 PM
> To: SIGMETRICS at LISTSERV.UTK.EDU
> Subject: Re: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez,
> and Herbert Van de Sompel "Journal Status"
> arXiv:cs.GL/0601030 v1 9 Jan 2006
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
>
>
>
>
> Loet,
> I see that you have once again taken my name in vain and
> again given me the opportunity to spout my ideas on
> SIGMETRICS.  I must admit that I have not read the paper you
> discuss, because my doctor warned me against reading too many
> such papers, since I am fairly close to OD-ing on them.
> However, the conclusions you mention do seem a little peculiar.
>
> Due to detailed study of Gene Garfield's development and
> utilization of impact factor, I am coming to change my mind
> on this measure somewhat.  It is for rather complicated
> reasons, which I shall try to explain below.
>
> In general I think that there is too much random error in
> citation data for the utilization of such precise techniques
> as correlation--Pearson, Spearman, whatever.  Much results
> from exogenous citations due to an inability to define
> precise sets--a logical consequence of Bradford's Law of
> Scattering and Garfield's Law of Concentration.  Impact
> factor suffers from a further source of error due to an
> inability to classify precisely sources into citable and
> non-citable--something which honest persons can disagree on.
> This inability severely affects the denominator of the impact
> factor equation.  What is therefore needed is a technique
> that is crude and robust against such error.  I have
> personally found it in the chi-square test of independence,
> which allows the conversion of citation measures into ordinal
> variables defined by broad categories.  It also allows one to
> define the amount of error one is willing to accept, i.e.,
> upper 10% vs.
> upper 25%.
>
> Use of this chi-square test may vindicate impact factor by
> demonstrating that it has the same strong relationship to
> expert ratings as do total citations.  As a matter of fact,
> it may be a superior measure in that it will not only capture
> the importance of reseach journals but also of review
> journals.  Close inspection of the top 10% of the journals
> recommended by the LSU chemistry faculty reveals it to be a
> balanced mix of research journals, review, journals, and the
> main teaching journal of chemistry.
> In other words, most facets of journal importance are
> captured by this measure, whereas total citations captures
> mainly research, and impact factor captures chiefly the
> review journals.  However, broadening the categories may
> cause impact factor to capture both research and review
> though not the teaching facet.  In any case I am going to
> test this in the revision of the JASIST paper I am now engaged in.
>
> Impact factor has the ability to do this for the very reasons
> Seglen denounces it.  His main case against is based on the
> reasoning of the law of error and the role of the arithmetic
> mean in this law.  This requires the normal distribution for
> the arithmetic mean to be an accurate estimate of central
> tendency.  However, due to the highly skewed distributions
> with which we deal, the arithmetic mean is always way above
> the other estimates of central tendency such as the median or
> the geometric mean due to the high degree of variance caused
> the dominant observations.  Seglen's reasoning collapses once
> one realizes that a journal's or scientist's importance is
> not measured by central tendency but by the variance caused
> by the few important articles published by the journal or scientist.
> Therefore, scientific importance is the result of variance
> and not central tendency.  The arithmetic mean, which impact
> factor attempts to estimate, better captures the variance.
>
> To demonstrate, I have converted Garfield's constant for the
> year 1993 into binomial p and the Poisson lambda  The way I
> did this is in the attached Excel file.  You will see the
> binomial p is a lousy 0.0003, which converts into a Poisson
> lambda or Garfield's constant of 2.15 for the year.  This is
> the probability or the rate articles were cited in 1993 on
> the assumption of probabilistic homogeneity.  However, since
> there is probabilistic heterogeneity, most articles have to
> have a citation rate below Garfield's constant.  True to
> form, of the 5000 journals covered that year, 4500 journals
> were below to Garfield's constant.  2.15 is an awful small
> range to squeeze 4500 journals into and expect meaningful
> quantitative distinctions.  Utilization of a central tendency
> measure puts one right smack in the middle of that tight
> range.  Small as this may be, the probabilities and lambda
> were actually much smaller, for Garfield's constant is based
> on the set of articles actually cited that year, i.e., it it
> truncated on the left and does not take into account the
> articles that could have been cited but were not.  I do not
> have the technical or intellectual ability to estimate this
> zero class.  I do know that Sir Maurice Kendall backed off
> from the problem when he confronted it in Bradford's Law, and
> who the hell am I compared to Maurice Kendall.  I wish that
> somebody would write an article understandable to simpletons
> on how to make such estimates.  From my perspective, this
> would be one of the most important articles ever written.
>
> Sorry for the tirade, but I thought I'd float a few trial
> balloons to be shot down.
>
> SB
>
> (See attached file: GarConst.xls)
>
>
>
>
>
>
>
>
> Loet Leydesdorff <loet at LEYDESDORFF.NET>@LISTSERV.UTK.EDU> on
> 03/04/2006
> 07:14:57 AM
>
> Please respond to ASIS&T Special Interest Group on Metrics
>        <SIGMETRICS at LISTSERV.UTK.EDU>
>
> Sent by:    ASIS&T Special Interest Group on Metrics
>        <SIGMETRICS at LISTSERV.UTK.EDU>
>
>
> To:    SIGMETRICS at LISTSERV.UTK.EDU
> cc:     (bcc: Stephen J Bensman/notsjb/LSU)
>
> Subject:    Re: [SIGMETRICS] Johan Bollen, Marko A.
> Rodriguez, and Herbert
>        Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1
> 9 Jan 2006
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Dear colleagues,
>
> The idea is interesting. However, there a few problems with
> this paper.
> First, the authors should not have used Pearson correlation
> coefficients to compare the rankings, but rank correlations
> (Spearman's rho or Kendall's tau). Second, it would have been
> interesting to have a rank correlation with "total cites"
> given recent discussions (Bensman). Third, the delineation of
> fields in terms of the ISI subject categories is very questionnable.
>
> However, the authors are very clear about their results: "We
> identified ...
> , but were unable to recognize a meaningful pattern in the
> results." (p.
> 9).
> I don't understand why one should then multiply the one
> measure with the other. What does multiplication to the error?
>
> Does one of you know a place where the ISI subject categories
> are justified?
> How are they produced? People seem to use them increasingly
> both in evaluation and research practices, but I have never
> been able to reproduce them using journal citation measures.
>
> With best wishes,
>
>
> Loet
>
>
> ________________________________
> Loet Leydesdorff
> Amsterdam School of Communications Research (ASCoR),
> Kloveniersburgwal 48, 1012 CX Amsterdam.
> Tel.: +31-20- 525 6598; fax: +31-20- 525 3681;
> loet at leydesdorff.net ; http://www.leydesdorff.net/
>
>
>
> > -----Original Message-----
> > From: ASIS&T Special Interest Group on Metrics
> > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Eugene Garfield
> > Sent: Friday, March 03, 2006 6:37 PM
> > To: SIGMETRICS at LISTSERV.UTK.EDU
> > Subject: [SIGMETRICS] Johan Bollen, Marko A. Rodriguez, and Herbert
> > Van de Sompel "Journal Status" arXiv:cs.GL/0601030 v1
> > 9 Jan 2006
> >
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> > Further to yesterday's posting, "Prestige is factored into journal
> > ratings", here is another interesting and informative article
> >
> > FULL TEXT AVAILABLE AT :
> > http://www.arxiv.org/PS_cache/cs/pdf/0601/0601030.pdf
> >
> > email: {jbollen, marko, herbertv}@lanl.gov
> >
> > TITLE   :  Journal Status
> >
> > AUTHORS :  Johan Bollen, Marko A. Rodriguez, and Herbert
> Van de Sompel
> >
> > SOURCE   : arXiv:cs.GL/0601030 v1 9 Jan 2006
> >
> > Abstract
> > The status of an actor in a social context is commonly defined in
> > terms of two factors: the total number of endorsements the actor
> > receives from other actors and the prestige of the
> endorsing actors.
> > These two factors indicate the distinction between popularity and
> > expert appreciation of the actor, respectively. We refer to
> the former
> > as popularity and to the latter as prestige. These notions of
> > popularity and prestige also apply to the domain of scholarly
> > assessment. The ISI Impact Factor (ISI IF) is defined as the mean
> > number of citations a journal receives over a 2 year
> period. By merely
> > counting the amount of citations and disregarding the
> prestige of the
> > citing journals, the ISI IF is a metric of popularity, not of
> > prestige. We demonstrate how a weighted version of the popular
> > PageRank algorithm can be used to obtain a metric that reflects
> > prestige. We contrast the rankings of journals according to
> their ISI
> > IF and their weighted PageRank, and we provide an analysis that
> > reveals both significant overlaps and differences.
> > Furthermore, we introduce the Y-factor which is a simple
> combination
> > of both the ISI IF and the weighted PageRank, and find that the
> > resulting journal rankings correspond well to a general
> understanding
> > of journal status.
> >
> >
> > ______________________________________________
> >
> >
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> > FULL TEXT AVAILABLE AT :
> >
> http://www.nature.com/nature/journal/v439/n7078/pdf/439770a.pdf    OR
> > http://guide.labanimal.com/news/2006/060213/full/439770a.html
> >
> >
> > Philip Ball : p.ball at nature.com
> > www.philipball.com
> >
> > Title: Prestige is factored into journal ratings
> >
> > Author(s): Ball P
> >
> > Source: NATURE 439 (7078): 770-771 FEB 16 2006
> >
> > Document Type: News Item Language: English
> > Cited References: 0      Times Cited: 0
> >
> > Publisher: NATURE PUBLISHING GROUP, MACMILLAN BUILDING, 4
> CRINAN ST,
> > LONDON
> > N1 9XW, ENGLAND
> > Subject Category: MULTIDISCIPLINARY SCIENCES IDS Number: 012JA
> >
> > ISSN: 0028-0836
> >
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20060306/608a1bfc/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: clip_image002.gif
Type: image/gif
Size: 3683 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20060306/608a1bfc/attachment.gif>