HistCite and RAEs
Stephen J Bensman
notsjb at LSU.EDU
Fri Apr 7 10:09:11 EDT 2006
Loet,
There is one point that I want to make that I should have made long ago but
somehow did not make the necessary connections. There is excellent
software that can perform very sophisticated program evaluations for RAEs.
It has all the necessary "metrics." It is HistCite that has been developed
by Sasha Pudovkin and Gene Garfield. Sasha gave me a lengthy demonstration
of this software at the ASIST Conference in Providence, RI, in 2004, and I
was mightly impressed. The program allows you first to define very
specific subject sets; it then picks out the key articles in these subject
sets and maps them with very informative graphics; it then allows you to do
institutional and national rankings. It compensates for all the faults of
the NRC meat axe approach. Moreover, since the UK and the US act as a
cultural unit, the correlations of ISI citations with British expert
ratings are very high, and I have proven the strong association of ISI
citations with British supralibrary use. Therefore, HistCite analyses
should conform to British concepts of program importance. From your
perspective, I suppose, the main fault is that it works on ISI subject
sets, but, in my opinion, these subject sets are about as good as can be
expected. One interesting experiment that HistCite might make possible to
test how a subject structure matches institutional structures. I am sure
that Gene Garfield would allow the UK government to have access to the
program for a reasonable price.
That said, there still remains one problem. Even with such a sophisticated
method of program evaluation, should the UK government allocate research
money on the basis of it, collectively punishing scientists not part of the
selected programs, or should the UK government remain neutral and allocate
money on the basis of the evaluation of individual projects? In either
case most of the money will go to the same programs, but at least the
others have a chance. Therefore, ideologically, I still favor the latter
approach.
SB
Loet Leydesdorff <loet at LEYDESDORFF.NET>@LISTSERV.UTK.EDU> on 04/06/2006
04:34:56 PM
Please respond to ASIS&T Special Interest Group on Metrics
<SIGMETRICS at LISTSERV.UTK.EDU>
Sent by: ASIS&T Special Interest Group on Metrics
<SIGMETRICS at LISTSERV.UTK.EDU>
To: SIGMETRICS at LISTSERV.UTK.EDU
cc: (bcc: Stephen J Bensman/notsjb/LSU)
Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
Dear Stephen,
Thanks for all this. It seems to me that we have exhaustively discussed the
RAE and the problems of replacing it with a metrics.
With best wishes,
Loet
________________________________
Loet Leydesdorff
Amsterdam School of Communications Research (ASCoR),
Kloveniersburgwal 48, 1012 CX Amsterdam.
Tel.: +31-20- 525 6598; fax: +31-20- 525 3681;
loet at leydesdorff.net ; http://www.leydesdorff.net/
> -----Original Message-----
> From: ASIS&T Special Interest Group on Metrics
> [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman
> Sent: Thursday, April 06, 2006 4:09 PM
> To: SIGMETRICS at LISTSERV.UTK.EDU
> Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Loet,
> We go back to the frequency theory of probability, which was
> best set forth by Richard Von Mises in his Wahr, Wahrheit,
> und Statistik (Truth, Probability, and Statistics). Von
> Mises states that no probability can calculated until a
> proper set--or "kollektiv" in his language--has been defined.
> Karl Pearson operated within the frequency theory, and he
> stated in his Grammar of Science that classification is the
> basis of all science, and he dismisses any study that is not
> based on classification as not science. So if you are going
> to rate programs, then you are going to have to establish
> precisely what it is you are going to rate and then select
> your data and measures accordingly. Take history for an
> example. You can rate history as a whole or select a
> historical specialty as southern history. Your selection of
> variables and bibliometric data will vary according to your
> purpose. By this very fact you are in a sense predetermining
> the outcome of who is going to come to the top. Now I know
> that the NRC was after history as a whole from a close up
> analysis of what they did. For bibliometric data they used
> the entire SSCI. Moreover, I know which professors at LSU
> were selected for the ratings. LSU has its strongest faculty
> in Southern history but these were not selected. Instead the
> professors in Russian and Chinese were selected for the
> ratings. This seems illogical, but then it has to be
> remembered that LSU has one of the few programs big enough to
> hate specialists in Russian and Chinese history, and these
> were needed for an adequate rating sample.
>
> Now I do not understand exactly how the Brits go about their
> RAEs. From what I understand, a prmgram has to volunteer to
> be rated, or otherwise it is not rated. So the sample seems
> to be self-selecting. The programs then submit publications
> to a committee, which uses them as a basis for ratings.
> But these programs may have different strengths and agendas,
> and this should affect the ratings. But do the ratings have
> the purpose of selecting some research areas as more
> important than other research areas?
> This I do not know, and this would definitely affect the
> ratings. Is this what you mean by "circular."
>
> What I would mean by "circular" is that, due to the stability
> of distributions, the same programs would be selected again
> and again for funding, reinforcing the hierarchy, and
> blocking either lesser programs or better faculty at the
> lesser programs from advancing their agendas.
>
> Did you make any sense out of all of this confusion?
>
> SB
>
>
>
>
>
>
> Loet Leydesdorff <loet at LEYDESDORFF.NET>@LISTSERV.UTK.EDU> on
> 04/06/2006
> 12:13:37 AM
>
> Please respond to ASIS&T Special Interest Group on Metrics
> <SIGMETRICS at LISTSERV.UTK.EDU>
>
> Sent by: ASIS&T Special Interest Group on Metrics
> <SIGMETRICS at LISTSERV.UTK.EDU>
>
>
> To: SIGMETRICS at LISTSERV.UTK.EDU
> cc: (bcc: Stephen J Bensman/notsjb/LSU)
>
> Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> If I correctly understand, you wish to say that any ranking
> of authors or institutions is ultimately dependent on how the
> sets are defined. A definition of sets on the basis of
> institutions would thus make the RAE operation circular.
>
> With best wishes, Loet
>
> ________________________________
> Loet Leydesdorff
> Amsterdam School of Communications Research (ASCoR),
> Kloveniersburgwal 48, 1012 CX Amsterdam.
> Tel.: +31-20- 525 6598; fax: +31-20- 525 3681;
> loet at leydesdorff.net ; http://www.leydesdorff.net/
>
>
>
> > -----Original Message-----
> > From: ASIS&T Special Interest Group on Metrics
> > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman
> > Sent: Wednesday, April 05, 2006 11:16 PM
> > To: SIGMETRICS at LISTSERV.UTK.EDU
> > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> >
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> > Loet,
> > The principle in real estate is "location, location,
> location." The
> > principle in program evaluation is "set definition, set definition,
> > set definition." I pointed out in another posting that a major
> > discovery of the 1993 NRC rating was that all previous
> ratings in the
> > biosciences were incorrect due to an incorrect method of
> > classification resulting in non-comparable sets. I am
> somewhat proud
> > that I was able to show to the NRC people how a change in
> > classification method had an enormous impact on the ratings of LSU,
> > turning us from a nonentity into something quite
> respectable and more
> > in line with Louisiana's pioneering role in medicine through mainly
> > the Ochsner Clinic and the first attempt at a charity
> hospital system.
> >
> > SB
> >
> >
> >
> >
> > Loet Leydesdorff <loet at LEYDESDORFF.NET>@LISTSERV.UTK.EDU> on
> > 03/31/2006
> > 11:57:46 AM
> >
> > Please respond to ASIS&T Special Interest Group on Metrics
> > <SIGMETRICS at LISTSERV.UTK.EDU>
> >
> > Sent by: ASIS&T Special Interest Group on Metrics
> > <SIGMETRICS at LISTSERV.UTK.EDU>
> >
> >
> > To: SIGMETRICS at LISTSERV.UTK.EDU
> > cc: (bcc: Stephen J Bensman/notsjb/LSU)
> >
> > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> >
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> > Dear Stephen,
> >
> > Although I am politically at the other end of the spectrum, I fully
> > agree with your critique of the RAE. But the critique would equally
> > hold for a "metric" that would rate departments against
> each other as
> > proposed by some of our colleagues. The problem is to take
> departments
> > as units of analysis.
> >
> > With best wishes,
> >
> >
> > Loet
> >
> > > -----Original Message-----
> > > From: ASIS&T Special Interest Group on Metrics
> > > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Stephen
> J Bensman
> > > Sent: Thursday, March 30, 2006 10:32 PM
> > > To: SIGMETRICS at listserv.utk.edu
> > > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> > >
> > > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > > http://web.utk.edu/~gwhitney/sigmetrics.html
> > >
> > > Gee, I consider myself anything but a cultural elitist.
> > > After all, I work at LSU. The basic problem of the RAE is
> > that it is
> > > biased against an institution like LSU. At least under the
> > American
> > > system, good researchers at a place like LSU have an even
> chance to
> > > obtain research funding, and many take advantage of this
> > system. That
> > > way a good researcher maintains his independence and advance his
> > > career. This way LSU plays a major role as a launch pad
> for up and
> > > coming scientists. The British RAE always reminded me of
> > the Tsarist
> > > system of krugovaia poruka, where all the peasants of a
> > commune were
> > > held liable for communal taxes. This was the taxation system of
> > > serfdom, causing peasants to be chained to the commune, stifling
> > > individual initiative, thereby causing agricultural
> stagnation, and
> > > ultimately a violent revolution.
> > > If this makes me a cultural elitist, then so be it.
> > >
> > > SB
> > >
> > >
> > >
> > >
> > > Phil Davis <pmd8 at CORNELL.EDU>@LISTSERV.UTK.EDU> on 03/30/2006
> > > 02:09:28 PM
> > >
> > > Please respond to ASIS&T Special Interest Group on Metrics
> > > <SIGMETRICS at LISTSERV.UTK.EDU>
> > >
> > > Sent by: ASIS&T Special Interest Group on Metrics
> > > <SIGMETRICS at LISTSERV.UTK.EDU>
> > >
> > >
> > > To: SIGMETRICS at LISTSERV.UTK.EDU
> > > cc: (bcc: Stephen J Bensman/notsjb/LSU)
> > >
> > > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> > >
> > > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > > http://web.utk.edu/~gwhitney/sigmetrics.html
> > >
> > > Stephen, I wouldn't call you a "capitalist pig" but a
> > willfully blind,
> > > cultural elitist. In countries where education is wholly
> > (or mostly)
> > > funded by the government -- not just the UK and Europe, but
> > Canada and
> > > others -- the government is concerned about making sure
> > that everyone
> > > gets some modicum of funding. That does not mean a completely
> > > equitable rationing system, but it ensures a base-level of
> > funding.
> > > In the United States, this base-level funding often comes
> > from one's
> > > own department or college. Granted, the
> > capitalist-approach you speak
> > > of does reward the best and greatest, and this Winner-takes-all
> > > approach does result in pioneering research, yet it only
> > rewards the
> > > few.
> > >
> > > --Phil Davis
> > >
> > >
> > >
> > > Stephen Bensman wrote:
> > >
> > > >Speaking as a capitalist pig, the entire RAE system is
> just another
> > > example
> > > >of socialists hoisting themselves on their own petards.
> > > Point 1 below
> > > >contains the essence of the problem. The US has done
> > > pioneering work
> > > >on the evaluation of research-doctorate programs but was
> > never silly
> > > >enough
> > > to
> > > >allocate research resources on the basis of it. Luckily
> > > because these
> > > >evaluations were usually screwed up in some way. Allocation of
> > > >research resources was always done on a project-by-project
> > > basis by the
> > > >NSF, NIH, and others, with experts in the fields evaluating
> > > individual
> > > >research proposals. The Europeans have a tendency to overplan
> > > >everything with disastrous consequences--the disaster in
> > > Eastern Europe
> > > >just being the latest example of it.
> > > >
> > > >SB
> > >
> >
>
More information about the SIGMETRICS
mailing list