Future UK RAEs to be Metrics-Based

[Loet Leydesdorff]

Dear Stephen,

Thanks for all this. It seems to me that we have exhaustively discussed the
RAE and the problems of replacing it with a metrics.

With best wishes,


Loet

________________________________
Loet Leydesdorff
Amsterdam School of Communications Research (ASCoR),
Kloveniersburgwal 48, 1012 CX Amsterdam.
Tel.: +31-20- 525 6598; fax: +31-20- 525 3681;
loet at leydesdorff.net ; http://www.leydesdorff.net/



> -----Original Message-----
> From: ASIS&T Special Interest Group on Metrics
> [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman
> Sent: Thursday, April 06, 2006 4:09 PM
> To: SIGMETRICS at LISTSERV.UTK.EDU
> Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Loet,
> We go back to the frequency theory of probability, which was
> best set forth by Richard Von Mises in his Wahr, Wahrheit,
> und Statistik (Truth, Probability, and Statistics).  Von
> Mises states that no probability can calculated until a
> proper set--or "kollektiv" in his language--has been defined.
>  Karl Pearson operated within the frequency theory, and he
> stated in his Grammar of Science that classification is the
> basis of all science, and he dismisses any study that is not
> based on classification as not science.  So if you are going
> to rate programs, then you are going to have to establish
> precisely what it is you are going to rate and then select
> your data and measures accordingly.  Take history for an
> example.  You can rate history as a whole or select a
> historical specialty as southern history.  Your selection of
> variables and bibliometric data will vary according to your
> purpose.  By this very fact you are in a sense predetermining
> the outcome of who is going to come to the top.  Now I know
> that the NRC was after history as a whole from a close up
> analysis of what they did.  For bibliometric data they used
> the entire SSCI.  Moreover, I know which professors at LSU
> were selected for the ratings.  LSU has its strongest faculty
> in Southern history but these were not selected.  Instead the
> professors in Russian and Chinese were selected for the
> ratings.  This seems illogical, but then it has to be
> remembered that LSU has one of the few programs big enough to
> hate specialists in Russian and Chinese history, and these
> were needed for an adequate rating sample.
>
> Now I do not understand exactly how the Brits go about their
> RAEs.  From what I understand, a prmgram has to volunteer to
> be rated, or otherwise it is not rated.  So the sample seems
> to be self-selecting.  The programs then submit publications
> to a committee, which uses them as a basis for ratings.
> But these programs may have different strengths and agendas,
> and this should affect the ratings.  But do the ratings have
> the purpose of selecting some research areas as more
> important than other research areas?
> This I do not know, and this would definitely affect the
> ratings.  Is this what you mean by "circular."
>
> What I would mean by "circular" is that, due to the stability
> of distributions, the same programs would be selected again
> and again for funding, reinforcing the hierarchy, and
> blocking either lesser programs or better faculty at the
> lesser programs from advancing their agendas.
>
> Did you make any sense out of all of this confusion?
>
> SB
>
>
>
>
>
>
> Loet Leydesdorff <loet at LEYDESDORFF.NET>@LISTSERV.UTK.EDU> on
> 04/06/2006
> 12:13:37 AM
>
> Please respond to ASIS&T Special Interest Group on Metrics
>        <SIGMETRICS at LISTSERV.UTK.EDU>
>
> Sent by:    ASIS&T Special Interest Group on Metrics
>        <SIGMETRICS at LISTSERV.UTK.EDU>
>
>
> To:    SIGMETRICS at LISTSERV.UTK.EDU
> cc:     (bcc: Stephen J Bensman/notsjb/LSU)
>
> Subject:    Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> If I correctly understand, you wish to say that any ranking
> of authors or institutions is ultimately dependent on how the
> sets are defined. A definition of sets on the basis of
> institutions would thus make the RAE operation circular.
>
> With best wishes,  Loet
>
> ________________________________
> Loet Leydesdorff
> Amsterdam School of Communications Research (ASCoR),
> Kloveniersburgwal 48, 1012 CX Amsterdam.
> Tel.: +31-20- 525 6598; fax: +31-20- 525 3681;
> loet at leydesdorff.net ; http://www.leydesdorff.net/
>
>
>
> > -----Original Message-----
> > From: ASIS&T Special Interest Group on Metrics
> > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman
> > Sent: Wednesday, April 05, 2006 11:16 PM
> > To: SIGMETRICS at LISTSERV.UTK.EDU
> > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> >
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> > Loet,
> > The principle in real estate is "location, location,
> location."  The
> > principle in program evaluation is "set definition, set definition,
> > set definition."  I pointed out in another posting that a major
> > discovery of the 1993 NRC rating was that all previous
> ratings in the
> > biosciences were incorrect due to an incorrect method of
> > classification resulting in non-comparable sets.  I am
> somewhat proud
> > that I was able to show to the NRC people how a change in
> > classification method had an enormous impact on the ratings of LSU,
> > turning us from a nonentity into something quite
> respectable and more
> > in line with Louisiana's pioneering role in medicine through mainly
> > the Ochsner Clinic and the first attempt at a charity
> hospital system.
> >
> > SB
> >
> >
> >
> >
> > Loet Leydesdorff <loet at LEYDESDORFF.NET>@LISTSERV.UTK.EDU> on
> > 03/31/2006
> > 11:57:46 AM
> >
> > Please respond to ASIS&T Special Interest Group on Metrics
> >        <SIGMETRICS at LISTSERV.UTK.EDU>
> >
> > Sent by:    ASIS&T Special Interest Group on Metrics
> >        <SIGMETRICS at LISTSERV.UTK.EDU>
> >
> >
> > To:    SIGMETRICS at LISTSERV.UTK.EDU
> > cc:     (bcc: Stephen J Bensman/notsjb/LSU)
> >
> > Subject:    Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> >
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> > Dear Stephen,
> >
> > Although I am politically at the other end of the spectrum, I fully
> > agree with your critique of the RAE. But the critique would equally
> > hold for a "metric" that would rate departments against
> each other as
> > proposed by some of our colleagues. The problem is to take
> departments
> > as units of analysis.
> >
> > With best wishes,
> >
> >
> > Loet
> >
> > > -----Original Message-----
> > > From: ASIS&T Special Interest Group on Metrics
> > > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Stephen
> J Bensman
> > > Sent: Thursday, March 30, 2006 10:32 PM
> > > To: SIGMETRICS at listserv.utk.edu
> > > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> > >
> > > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > > http://web.utk.edu/~gwhitney/sigmetrics.html
> > >
> > > Gee, I consider myself anything but a cultural elitist.
> > > After all, I work at LSU.  The basic problem of the RAE is
> > that it is
> > > biased against an institution like LSU.  At least under the
> > American
> > > system, good researchers at a place like LSU have an even
> chance to
> > > obtain research funding, and many take advantage of this
> > system.  That
> > > way a good researcher maintains his independence and advance his
> > > career.  This way LSU plays a major role as a launch pad
> for up and
> > > coming scientists.  The British RAE always reminded me of
> > the Tsarist
> > > system of krugovaia poruka, where all the peasants of a
> > commune were
> > > held liable for communal taxes.  This was the taxation system of
> > > serfdom, causing peasants to be chained to the commune, stifling
> > > individual initiative, thereby causing agricultural
> stagnation, and
> > > ultimately a violent revolution.
> > >  If this makes me a cultural elitist, then so be it.
> > >
> > > SB
> > >
> > >
> > >
> > >
> > > Phil Davis <pmd8 at CORNELL.EDU>@LISTSERV.UTK.EDU> on 03/30/2006
> > > 02:09:28 PM
> > >
> > > Please respond to ASIS&T Special Interest Group on Metrics
> > >        <SIGMETRICS at LISTSERV.UTK.EDU>
> > >
> > > Sent by:    ASIS&T Special Interest Group on Metrics
> > >        <SIGMETRICS at LISTSERV.UTK.EDU>
> > >
> > >
> > > To:    SIGMETRICS at LISTSERV.UTK.EDU
> > > cc:     (bcc: Stephen J Bensman/notsjb/LSU)
> > >
> > > Subject:    Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> > >
> > > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > > http://web.utk.edu/~gwhitney/sigmetrics.html
> > >
> > > Stephen, I wouldn't call you a "capitalist pig" but a
> > willfully blind,
> > > cultural elitist.  In countries where education is wholly
> > (or mostly)
> > > funded by the government -- not just the UK and Europe, but
> > Canada and
> > > others -- the government is concerned about making sure
> > that everyone
> > > gets some modicum of funding.  That does not mean a completely
> > > equitable rationing system, but it ensures a base-level of
> > funding.
> > > In the United States, this base-level funding often comes
> > from one's
> > > own department or college.  Granted, the
> > capitalist-approach you speak
> > > of does reward the best and greatest, and this Winner-takes-all
> > > approach does result in pioneering research, yet it only
> > rewards the
> > > few.
> > >
> > > --Phil Davis
> > >
> > >
> > >
> > > Stephen Bensman wrote:
> > >
> > > >Speaking as a capitalist pig, the entire RAE system is
> just another
> > > example
> > > >of socialists hoisting themselves on their own petards.
> > > Point 1 below
> > > >contains the essence of the problem.  The US has done
> > > pioneering work
> > > >on the evaluation of research-doctorate programs but was
> > never silly
> > > >enough
> > > to
> > > >allocate research resources on the basis of it.  Luckily
> > > because these
> > > >evaluations were usually screwed up in some way.  Allocation of
> > > >research resources was always done on a project-by-project
> > > basis by the
> > > >NSF, NIH, and others, with experts in the fields evaluating
> > > individual
> > > >research proposals.  The Europeans have a tendency to overplan
> > > >everything with disastrous consequences--the disaster in
> > > Eastern Europe
> > > >just being the latest example of it.
> > > >
> > > >SB
> > >
> >
>