Future UK RAEs to be Metrics-Based
Stephen J Bensman
notsjb at LSU.EDU
Thu Apr 6 10:08:34 EDT 2006
Loet,
We go back to the frequency theory of probability, which was best set forth
by Richard Von Mises in his Wahr, Wahrheit, und Statistik (Truth,
Probability, and Statistics). Von Mises states that no probability can
calculated until a proper set--or "kollektiv" in his language--has been
defined. Karl Pearson operated within the frequency theory, and he stated
in his Grammar of Science that classification is the basis of all science,
and he dismisses any study that is not based on classification as not
science. So if you are going to rate programs, then you are going to have
to establish precisely what it is you are going to rate and then select
your data and measures accordingly. Take history for an example. You can
rate history as a whole or select a historical specialty as southern
history. Your selection of variables and bibliometric data will vary
according to your purpose. By this very fact you are in a sense
predetermining the outcome of who is going to come to the top. Now I know
that the NRC was after history as a whole from a close up analysis of what
they did. For bibliometric data they used the entire SSCI. Moreover, I
know which professors at LSU were selected for the ratings. LSU has its
strongest faculty in Southern history but these were not selected. Instead
the professors in Russian and Chinese were selected for the ratings. This
seems illogical, but then it has to be remembered that LSU has one of the
few programs big enough to hate specialists in Russian and Chinese history,
and these were needed for an adequate rating sample.
Now I do not understand exactly how the Brits go about their RAEs. From
what I understand, a prmgram has to volunteer to be rated, or otherwise it
is not rated. So the sample seems to be self-selecting. The programs then
submit publications to a committee, which uses them as a basis for ratings.
But these programs may have different strengths and agendas, and this
should affect the ratings. But do the ratings have the purpose of
selecting some research areas as more important than other research areas?
This I do not know, and this would definitely affect the ratings. Is this
what you mean by "circular."
What I would mean by "circular" is that, due to the stability of
distributions, the same programs would be selected again and again for
funding, reinforcing the hierarchy, and blocking either lesser programs or
better faculty at the lesser programs from advancing their agendas.
Did you make any sense out of all of this confusion?
SB
Loet Leydesdorff <loet at LEYDESDORFF.NET>@LISTSERV.UTK.EDU> on 04/06/2006
12:13:37 AM
Please respond to ASIS&T Special Interest Group on Metrics
<SIGMETRICS at LISTSERV.UTK.EDU>
Sent by: ASIS&T Special Interest Group on Metrics
<SIGMETRICS at LISTSERV.UTK.EDU>
To: SIGMETRICS at LISTSERV.UTK.EDU
cc: (bcc: Stephen J Bensman/notsjb/LSU)
Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
If I correctly understand, you wish to say that any ranking of authors or
institutions is ultimately dependent on how the sets are defined. A
definition of sets on the basis of institutions would thus make the RAE
operation circular.
With best wishes, Loet
________________________________
Loet Leydesdorff
Amsterdam School of Communications Research (ASCoR),
Kloveniersburgwal 48, 1012 CX Amsterdam.
Tel.: +31-20- 525 6598; fax: +31-20- 525 3681;
loet at leydesdorff.net ; http://www.leydesdorff.net/
> -----Original Message-----
> From: ASIS&T Special Interest Group on Metrics
> [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stephen J Bensman
> Sent: Wednesday, April 05, 2006 11:16 PM
> To: SIGMETRICS at LISTSERV.UTK.EDU
> Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Loet,
> The principle in real estate is "location, location,
> location." The principle in program evaluation is "set
> definition, set definition, set definition." I pointed out
> in another posting that a major discovery of the 1993 NRC
> rating was that all previous ratings in the biosciences were
> incorrect due to an incorrect method of classification
> resulting in non-comparable sets. I am somewhat proud that I
> was able to show to the NRC people how a change in
> classification method had an enormous impact on the ratings
> of LSU, turning us from a nonentity into something quite
> respectable and more in line with Louisiana's pioneering role
> in medicine through mainly the Ochsner Clinic and the first
> attempt at a charity hospital system.
>
> SB
>
>
>
>
> Loet Leydesdorff <loet at LEYDESDORFF.NET>@LISTSERV.UTK.EDU> on
> 03/31/2006
> 11:57:46 AM
>
> Please respond to ASIS&T Special Interest Group on Metrics
> <SIGMETRICS at LISTSERV.UTK.EDU>
>
> Sent by: ASIS&T Special Interest Group on Metrics
> <SIGMETRICS at LISTSERV.UTK.EDU>
>
>
> To: SIGMETRICS at LISTSERV.UTK.EDU
> cc: (bcc: Stephen J Bensman/notsjb/LSU)
>
> Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Dear Stephen,
>
> Although I am politically at the other end of the spectrum, I
> fully agree with your critique of the RAE. But the critique
> would equally hold for a "metric" that would rate departments
> against each other as proposed by some of our colleagues. The
> problem is to take departments as units of analysis.
>
> With best wishes,
>
>
> Loet
>
> > -----Original Message-----
> > From: ASIS&T Special Interest Group on Metrics
> > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Stephen J Bensman
> > Sent: Thursday, March 30, 2006 10:32 PM
> > To: SIGMETRICS at listserv.utk.edu
> > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> >
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> > Gee, I consider myself anything but a cultural elitist.
> > After all, I work at LSU. The basic problem of the RAE is
> that it is
> > biased against an institution like LSU. At least under the
> American
> > system, good researchers at a place like LSU have an even chance to
> > obtain research funding, and many take advantage of this
> system. That
> > way a good researcher maintains his independence and advance his
> > career. This way LSU plays a major role as a launch pad for up and
> > coming scientists. The British RAE always reminded me of
> the Tsarist
> > system of krugovaia poruka, where all the peasants of a
> commune were
> > held liable for communal taxes. This was the taxation system of
> > serfdom, causing peasants to be chained to the commune, stifling
> > individual initiative, thereby causing agricultural stagnation, and
> > ultimately a violent revolution.
> > If this makes me a cultural elitist, then so be it.
> >
> > SB
> >
> >
> >
> >
> > Phil Davis <pmd8 at CORNELL.EDU>@LISTSERV.UTK.EDU> on 03/30/2006
> > 02:09:28 PM
> >
> > Please respond to ASIS&T Special Interest Group on Metrics
> > <SIGMETRICS at LISTSERV.UTK.EDU>
> >
> > Sent by: ASIS&T Special Interest Group on Metrics
> > <SIGMETRICS at LISTSERV.UTK.EDU>
> >
> >
> > To: SIGMETRICS at LISTSERV.UTK.EDU
> > cc: (bcc: Stephen J Bensman/notsjb/LSU)
> >
> > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based
> >
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> > Stephen, I wouldn't call you a "capitalist pig" but a
> willfully blind,
> > cultural elitist. In countries where education is wholly
> (or mostly)
> > funded by the government -- not just the UK and Europe, but
> Canada and
> > others -- the government is concerned about making sure
> that everyone
> > gets some modicum of funding. That does not mean a completely
> > equitable rationing system, but it ensures a base-level of
> funding.
> > In the United States, this base-level funding often comes
> from one's
> > own department or college. Granted, the
> capitalist-approach you speak
> > of does reward the best and greatest, and this Winner-takes-all
> > approach does result in pioneering research, yet it only
> rewards the
> > few.
> >
> > --Phil Davis
> >
> >
> >
> > Stephen Bensman wrote:
> >
> > >Speaking as a capitalist pig, the entire RAE system is just another
> > example
> > >of socialists hoisting themselves on their own petards.
> > Point 1 below
> > >contains the essence of the problem. The US has done
> > pioneering work
> > >on the evaluation of research-doctorate programs but was
> never silly
> > >enough
> > to
> > >allocate research resources on the basis of it. Luckily
> > because these
> > >evaluations were usually screwed up in some way. Allocation of
> > >research resources was always done on a project-by-project
> > basis by the
> > >NSF, NIH, and others, with experts in the fields evaluating
> > individual
> > >research proposals. The Europeans have a tendency to overplan
> > >everything with disastrous consequences--the disaster in
> > Eastern Europe
> > >just being the latest example of it.
> > >
> > >SB
> >
>
More information about the SIGMETRICS
mailing list