HistCite and RAEs
Loet Leydesdorff
loet at LEYDESDORFF.NET
Fri Apr 7 10:53:16 EDT 2006
So, there we are! Some of us seem to have the metrics on the shelf.
Congratulation to Sasha Pudovkin, Eugene Garfield, and Steven Harnad!
Unless, Stephen, you happen to have shares in the ISI? :-)
I was at the meeting in Prudence and got the same demonstration, but I
obviously did not see what you saw. Is there a publication which you would
advice, to understand these correlations?
With best wishes, Loet
________________________________
Loet Leydesdorff
Amsterdam School of Communications Research (ASCoR),
Kloveniersburgwal 48, 1012 CX Amsterdam.
Tel.: +31-20- 525 6598; fax: +31-20- 525 3681;
loet at leydesdorff.net ; http://www.leydesdorff.net/
> -----Original Message-----
> From: ASIS&T Special Interest Group on Metrics
> [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman
> Sent: Friday, April 07, 2006 4:09 PM
> To: SIGMETRICS at LISTSERV.UTK.EDU
> Subject: [SIGMETRICS] HistCite and RAEs
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Loet,
> There is one point that I want to make that I should have
> made long ago but somehow did not make the necessary
> connections. There is excellent software that can perform
> very sophisticated program evaluations for RAEs.
> It has all the necessary "metrics." It is HistCite that has
> been developed by Sasha Pudovkin and Gene Garfield. Sasha
> gave me a lengthy demonstration of this software at the ASIST
> Conference in Providence, RI, in 2004, and I was mightly
> impressed. The program allows you first to define very
> specific subject sets; it then picks out the key articles in
> these subject sets and maps them with very informative
> graphics; it then allows you to do institutional and national
> rankings. It compensates for all the faults of the NRC meat
> axe approach. Moreover, since the UK and the US act as a
> cultural unit, the correlations of ISI citations with British
> expert ratings are very high, and I have proven the strong
> association of ISI citations with British supralibrary use.
> Therefore, HistCite analyses should conform to British
> concepts of program importance. From your perspective, I
> suppose, the main fault is that it works on ISI subject sets,
> but, in my opinion, these subject sets are about as good as
> can be expected. One interesting experiment that HistCite
> might make possible to test how a subject structure matches
> institutional structures. I am sure that Gene Garfield would
> allow the UK government to have access to the program for a
> reasonable price.
>
> That said, there still remains one problem. Even with such a
> sophisticated method of program evaluation, should the UK
> government allocate research money on the basis of it,
> collectively punishing scientists not part of the selected
> programs, or should the UK government remain neutral and
> allocate money on the basis of the evaluation of individual
> projects? In either case most of the money will go to the
> same programs, but at least the others have a chance.
> Therefore, ideologically, I still favor the latter approach.
>
> SB
>
>
>
>
>
>
> Loet Leydesdorff <loet at LEYDESDORFF.NET>@LISTSERV.UTK.EDU> on
> 04/06/2006
> 04:34:56 PM
>
> Please respond to ASIS&T Special Interest Group on Metrics
> <SIGMETRICS at LISTSERV.UTK.EDU>
>
> Sent by: ASIS&T Special Interest Group on Metrics
> <SIGMETRICS at LISTSERV.UTK.EDU>
>
>
> To: SIGMETRICS at LISTSERV.UTK.EDU
> cc: (bcc: Stephen J Bensman/notsjb/LSU)
>
> Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Dear Stephen,
>
> Thanks for all this. It seems to me that we have exhaustively
> discussed the RAE and the problems of replacing it with a metrics.
>
> With best wishes,
>
>
> Loet
>
> ________________________________
> Loet Leydesdorff
> Amsterdam School of Communications Research (ASCoR),
> Kloveniersburgwal 48, 1012 CX Amsterdam.
> Tel.: +31-20- 525 6598; fax: +31-20- 525 3681;
> loet at leydesdorff.net ; http://www.leydesdorff.net/
>
>
>
> > -----Original Message-----
> > From: ASIS&T Special Interest Group on Metrics
> > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman
> > Sent: Thursday, April 06, 2006 4:09 PM
> > To: SIGMETRICS at LISTSERV.UTK.EDU
> > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> >
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> > Loet,
> > We go back to the frequency theory of probability, which
> was best set
> > forth by Richard Von Mises in his Wahr, Wahrheit, und Statistik
> > (Truth, Probability, and Statistics). Von Mises states that no
> > probability can calculated until a proper set--or
> "kollektiv" in his
> > language--has been defined.
> > Karl Pearson operated within the frequency theory, and he
> stated in
> > his Grammar of Science that classification is the basis of all
> > science, and he dismisses any study that is not based on
> > classification as not science. So if you are going to rate
> programs,
> > then you are going to have to establish precisely what it
> is you are
> > going to rate and then select your data and measures accordingly.
> > Take history for an example. You can rate history as a whole or
> > select a historical specialty as southern history. Your
> selection of
> > variables and bibliometric data will vary according to your
> purpose.
> > By this very fact you are in a sense predetermining the
> outcome of who
> > is going to come to the top. Now I know that the NRC was after
> > history as a whole from a close up analysis of what they did. For
> > bibliometric data they used the entire SSCI. Moreover, I
> know which
> > professors at LSU were selected for the ratings. LSU has its
> > strongest faculty in Southern history but these were not selected.
> > Instead the professors in Russian and Chinese were selected for the
> > ratings. This seems illogical, but then it has to be
> remembered that
> > LSU has one of the few programs big enough to hate specialists in
> > Russian and Chinese history, and these were needed for an adequate
> > rating sample.
> >
> > Now I do not understand exactly how the Brits go about their RAEs.
> > From what I understand, a prmgram has to volunteer to be rated, or
> > otherwise it is not rated. So the sample seems to be
> self-selecting.
> > The programs then submit publications to a committee, which
> uses them
> > as a basis for ratings.
> > But these programs may have different strengths and
> agendas, and this
> > should affect the ratings. But do the ratings have the purpose of
> > selecting some research areas as more important than other research
> > areas?
> > This I do not know, and this would definitely affect the
> ratings. Is
> > this what you mean by "circular."
> >
> > What I would mean by "circular" is that, due to the stability of
> > distributions, the same programs would be selected again
> and again for
> > funding, reinforcing the hierarchy, and blocking either lesser
> > programs or better faculty at the lesser programs from
> advancing their
> > agendas.
> >
> > Did you make any sense out of all of this confusion?
> >
> > SB
> >
> >
> >
> >
> >
> >
> > Loet Leydesdorff <loet at LEYDESDORFF.NET>@LISTSERV.UTK.EDU> on
> > 04/06/2006
> > 12:13:37 AM
> >
> > Please respond to ASIS&T Special Interest Group on Metrics
> > <SIGMETRICS at LISTSERV.UTK.EDU>
> >
> > Sent by: ASIS&T Special Interest Group on Metrics
> > <SIGMETRICS at LISTSERV.UTK.EDU>
> >
> >
> > To: SIGMETRICS at LISTSERV.UTK.EDU
> > cc: (bcc: Stephen J Bensman/notsjb/LSU)
> >
> > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> >
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> > If I correctly understand, you wish to say that any ranking
> of authors
> > or institutions is ultimately dependent on how the sets are
> defined. A
> > definition of sets on the basis of institutions would thus make the
> > RAE operation circular.
> >
> > With best wishes, Loet
> >
> > ________________________________
> > Loet Leydesdorff
> > Amsterdam School of Communications Research (ASCoR),
> Kloveniersburgwal
> > 48, 1012 CX Amsterdam.
> > Tel.: +31-20- 525 6598; fax: +31-20- 525 3681;
> loet at leydesdorff.net ;
> > http://www.leydesdorff.net/
> >
> >
> >
> > > -----Original Message-----
> > > From: ASIS&T Special Interest Group on Metrics
> > > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen
> J Bensman
> > > Sent: Wednesday, April 05, 2006 11:16 PM
> > > To: SIGMETRICS at LISTSERV.UTK.EDU
> > > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> > >
> > > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > > http://web.utk.edu/~gwhitney/sigmetrics.html
> > >
> > > Loet,
> > > The principle in real estate is "location, location,
> > location." The
> > > principle in program evaluation is "set definition, set
> definition,
> > > set definition." I pointed out in another posting that a major
> > > discovery of the 1993 NRC rating was that all previous
> > ratings in the
> > > biosciences were incorrect due to an incorrect method of
> > > classification resulting in non-comparable sets. I am
> > somewhat proud
> > > that I was able to show to the NRC people how a change in
> > > classification method had an enormous impact on the
> ratings of LSU,
> > > turning us from a nonentity into something quite
> > respectable and more
> > > in line with Louisiana's pioneering role in medicine
> through mainly
> > > the Ochsner Clinic and the first attempt at a charity
> > hospital system.
> > >
> > > SB
> > >
> > >
> > >
> > >
> > > Loet Leydesdorff <loet at LEYDESDORFF.NET>@LISTSERV.UTK.EDU> on
> > > 03/31/2006
> > > 11:57:46 AM
> > >
> > > Please respond to ASIS&T Special Interest Group on Metrics
> > > <SIGMETRICS at LISTSERV.UTK.EDU>
> > >
> > > Sent by: ASIS&T Special Interest Group on Metrics
> > > <SIGMETRICS at LISTSERV.UTK.EDU>
> > >
> > >
> > > To: SIGMETRICS at LISTSERV.UTK.EDU
> > > cc: (bcc: Stephen J Bensman/notsjb/LSU)
> > >
> > > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> > >
> > > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > > http://web.utk.edu/~gwhitney/sigmetrics.html
> > >
> > > Dear Stephen,
> > >
> > > Although I am politically at the other end of the
> spectrum, I fully
> > > agree with your critique of the RAE. But the critique
> would equally
> > > hold for a "metric" that would rate departments against
> > each other as
> > > proposed by some of our colleagues. The problem is to take
> > departments
> > > as units of analysis.
> > >
> > > With best wishes,
> > >
> > >
> > > Loet
> > >
> > > > -----Original Message-----
> > > > From: ASIS&T Special Interest Group on Metrics
> > > > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Stephen
> > J Bensman
> > > > Sent: Thursday, March 30, 2006 10:32 PM
> > > > To: SIGMETRICS at listserv.utk.edu
> > > > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> > > >
> > > > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > > > http://web.utk.edu/~gwhitney/sigmetrics.html
> > > >
> > > > Gee, I consider myself anything but a cultural elitist.
> > > > After all, I work at LSU. The basic problem of the RAE is
> > > that it is
> > > > biased against an institution like LSU. At least under the
> > > American
> > > > system, good researchers at a place like LSU have an even
> > chance to
> > > > obtain research funding, and many take advantage of this
> > > system. That
> > > > way a good researcher maintains his independence and
> advance his
> > > > career. This way LSU plays a major role as a launch pad
> > for up and
> > > > coming scientists. The British RAE always reminded me of
> > > the Tsarist
> > > > system of krugovaia poruka, where all the peasants of a
> > > commune were
> > > > held liable for communal taxes. This was the taxation
> system of
> > > > serfdom, causing peasants to be chained to the commune,
> stifling
> > > > individual initiative, thereby causing agricultural
> > stagnation, and
> > > > ultimately a violent revolution.
> > > > If this makes me a cultural elitist, then so be it.
> > > >
> > > > SB
> > > >
> > > >
> > > >
> > > >
> > > > Phil Davis <pmd8 at CORNELL.EDU>@LISTSERV.UTK.EDU> on 03/30/2006
> > > > 02:09:28 PM
> > > >
> > > > Please respond to ASIS&T Special Interest Group on Metrics
> > > > <SIGMETRICS at LISTSERV.UTK.EDU>
> > > >
> > > > Sent by: ASIS&T Special Interest Group on Metrics
> > > > <SIGMETRICS at LISTSERV.UTK.EDU>
> > > >
> > > >
> > > > To: SIGMETRICS at LISTSERV.UTK.EDU
> > > > cc: (bcc: Stephen J Bensman/notsjb/LSU)
> > > >
> > > > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> > > >
> > > > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > > > http://web.utk.edu/~gwhitney/sigmetrics.html
> > > >
> > > > Stephen, I wouldn't call you a "capitalist pig" but a
> > > willfully blind,
> > > > cultural elitist. In countries where education is wholly
> > > (or mostly)
> > > > funded by the government -- not just the UK and Europe, but
> > > Canada and
> > > > others -- the government is concerned about making sure
> > > that everyone
> > > > gets some modicum of funding. That does not mean a completely
> > > > equitable rationing system, but it ensures a base-level of
> > > funding.
> > > > In the United States, this base-level funding often comes
> > > from one's
> > > > own department or college. Granted, the
> > > capitalist-approach you speak
> > > > of does reward the best and greatest, and this Winner-takes-all
> > > > approach does result in pioneering research, yet it only
> > > rewards the
> > > > few.
> > > >
> > > > --Phil Davis
> > > >
> > > >
> > > >
> > > > Stephen Bensman wrote:
> > > >
> > > > >Speaking as a capitalist pig, the entire RAE system is
> > just another
> > > > example
> > > > >of socialists hoisting themselves on their own petards.
> > > > Point 1 below
> > > > >contains the essence of the problem. The US has done
> > > > pioneering work
> > > > >on the evaluation of research-doctorate programs but was
> > > never silly
> > > > >enough
> > > > to
> > > > >allocate research resources on the basis of it. Luckily
> > > > because these
> > > > >evaluations were usually screwed up in some way.
> Allocation of
> > > > >research resources was always done on a project-by-project
> > > > basis by the
> > > > >NSF, NIH, and others, with experts in the fields evaluating
> > > > individual
> > > > >research proposals. The Europeans have a tendency to overplan
> > > > >everything with disastrous consequences--the disaster in
> > > > Eastern Europe
> > > > >just being the latest example of it.
> > > > >
> > > > >SB
> > > >
> > >
> >
>
More information about the SIGMETRICS
mailing list