RAE Questions

[David Goodman]

Dear Stephen,

You report that NRC ratings
> correlated from one period to another about at 0.9.
but you object to a departmental based distribution of research
funds because
> Analysis of the
> elite programs reveals a surprising amount of dead wood  among their
> scientists

If both statements are correct, then the only way the NRC ratings can be
valid is if the  presence of a few such people in a department
does not actually harm the department.  The difficulty  with  the
RAE approach is then "only" the problem of equitably distributing
resources within a department.
To a certain extent this is done by the students and post-
doctoral fellows: they will work only with the active members.

Dr. David Goodman
Associate Professor
Palmer School of Library and Information Science
Long Island University
and formerly
Princeton University Library

dgoodman at liu.edu
dgoodman at princeton.edu

----- Original Message -----
From: Stephen J Bensman <notsjb at LSU.EDU>
Date: Monday, April 3, 2006 2:23 pm
Subject: [SIGMETRICS] RAE Questions
To: SIGMETRICS at listserv.utk.edu

> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> I would like to use the commentary of David Watkins below to raise
> somequestions I have concerning the British RAE.  I have some
> insight on this
> matter, for I worked closely with the National Research Council
> (NRC) on
> its last evaluation of US research-doctorate programs.  I was
> chosen as one
> of three persons to test the database which the NRC developed
> during this
> evalution.  The NRC data is some of the best data in world with
> which to
> analyze variables involved in academic evaluations, and it is freely
> available at the following web site for those interested:
>
> http://www.nap.edu/readingroom/books/researchdoc/
>
> Unfortunately it is not the full data set but only that published
> in the
> book, and you have to buy the book to understand it.
>
> My questions concern the effect of the types of distributions with
> whichthe RAEs are dealing.  It is well known that informetric
> distributions are
> not only highly skewed but highly stable over time.  Using the NRC
> data I
> found that peer rating, publication rates, citation rates, etc.
> correlatedfrom one period to another about at 0.9.  Moreover, the
> same programs
> maintained their dominant positions despite the addition of
> numerous new
> programs.  This is a natural consequence of the cumulative
> advantage or
> success-breeds-success process underlying this stratification system.
> Translating this into UK terms, it means that Oxford and Cambridge
> havebeen dominant for about the last millenium and will maintain their
> dominance for the next millenium, barring their conversion into
> madrassas.The RAES should not only be finding this but reinforcing it.
>
> Second, such a hierarchical and stable social system is probably
> functionally necessary for the advance of science.  Then comes the
> questionabout what is the mobility of individual scientists within
> this system.
> Here I think may be the primary fault of the RAEs.  The
> distributions are
> not only skewed between programs but within programs.  Analysis of the
> elite programs reveals a surprising amount of dead wood  among their
> scientists.  Some of it results of extinct volcanos that have ceased
> producing new work decades ago.  The way the RAE allocates research
> resources seems to have the potential of feather-bedding these
> drones at
> the expense of the productive scientists at lesser institutions,
> making the
> system not only hierarchical but closed.
>
> And, third, evaluating research on a departmental basis seems to
> run the
> risk of introducing variables that extraneous to the quality of the
> research.  It is well known that large departments are
> automatically more
> highly rated than small ones just on that basis alone.  Moreover,
> such a
> broad method of evaluating research reduces the flexibility in
> definingproper sets for comparative purposes.  Thus, there is the
> danger of losing
> sight of new and developing fields.
>
> Taking the above points all together, I think that the US
> utilization of
> such evaluations is more correct than that of the UK.  In the US such
> evaluations are regarded as what they actually are--beauty contests in
> which academic programs strut down the runway in their swim suits,
> flaunting their particulars.  ("And, Miss Harvard Physics, how do
> you think
> the world can be immediately saved?")  The data can be used to
> determinehow the system is actually functioning and thus explore
> ways in which the
> system can be proved.  For example, I was able to use the NRC data to
> determine that there is no anti-Southern bias in the evaluation of
> research-doctorate programs, thus breaking numerous faculty hearts
> here at
> LSU.  However, such evaluations should in no way be utilized for
> allocatingresearch funds, which is best done on a project-by-
> project basis.
> Fortunately, the UK way is politically impossible in the US, which
> is a
> Federal state.  The South may have lost the Civil War, but its
> soldiersinflicted 9 casualties for every one suffered, putting
> strict limits on the
> central Leviathan due to the huge butcher's bill.  Any attempt to
> implementthe UK system would be immediately killed in Congress and
> probably by the
> Southern delegations.
>
> These musings are only speculative, and I may be entirely wrong
> about the
> dangers of the UK system.
>
> SB
>
>
>
>
> David Watkins <David.Watkins at SOLENT.AC.UK>@LISTSERV.UTK.EDU> on
> 04/02/200608:32:03 AM
>
> Please respond to ASIS&T Special Interest Group on Metrics
>       <SIGMETRICS at LISTSERV.UTK.EDU>
>
> Sent by:    ASIS&T Special Interest Group on Metrics
>       <SIGMETRICS at LISTSERV.UTK.EDU>
>
>
> To:    SIGMETRICS at LISTSERV.UTK.EDU
> cc:     (bcc: Stephen J Bensman/notsjb/LSU)
>
> Subject:    Re: [SIGMETRICS] SIGMETRICS Digest - 30 Mar 2006 to 31
> Mar 2006
>       (#2006-44)
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> RAE et al.....
>
>
> For me the basic problem with the RAE was an unintended
> consequence. The
> government's problem was that of 'big science'. HEFCE's desire was to
> concentrate funding for expensive, scientific / medical / engineering
> research in a few institutions. Probably a good idea. However, it
> swept up
> all the other disciplines, too, into a one size fits all system.
> Thus the
> lone researcher who can do good work in the history stacks and local
> archives - if given sufficient time - was treated like the guy who
> needsCERN time and a cast of hundreds to check out his/her theory.
> That is just
> bizarre. In the humanities and social sciences the results have
> been poor
> for academe since the basic premise was wrong.
>
> Switching the whole system to a metrics based one merely continues
> this and
> distorts scholarly good practice outside the sciences; we know that
> publication practices, citation behaviour etc are quite different in
> non-science disciplines in the absence of a distortion like RAE.
> What is
> needed is a funding regime which gives scholarly space-time and
> base-line
> funding to all academics - possibly working on the basis of a
> notionallyequal 4 way split between teaching/teaching
> preparation/administration/research. Those who needed no special
> facilitiescould do good work on this basis as they always did
> traditionally.The rest
> of HEFCE's research funds would go to the Research Councils. Other
> academics who needed large scale funding / facilities would use their
> 'quartile' to work up proposals for substantial funding from the
> ResearchCouncils and others. This could - but needn't  - involve
> buying out more of
> their own admin - or even teaching - time. Teaching everywhere
> would be
> enhanced since few established academics would have / need non-
> teachingroles (evaluation of the whole scholarly role internally,
> with a level
> playing field) and the costs of external evaluation would be
> restricted to
> the project level, which is the most appropriate one. There would
> be no
> institutional 'halo' to support underperformers, but good scholars
> workingin isolation or in small / peripheral institutions would not be
> discriminated against, and recruitment could return to the
> assessment of
> academic institutional need rather than just looking for 'four alpha
> publications' and to hell with the other academic skills and
> interests we
> would all favour seeing in a new colleague.
>
>
> ************************************************
> Professor David Watkins
> Postgraduate Research Centre
> Southampton Business School
> East Park Terrace
> Southampton SO14 0RH
>
> David.Watkins at solent.ac.uk
> 023 80 319610 (Tel)
> +44 23 80 31 96 10 (Tel)
>
> 02380 33 26 27 (fax)
> +44 23 80 33 26 27 (fax)
>